2023-05-18 03:53:36,801 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-05-18 03:53:37,100 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-05-18 03:53:37,199 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-05-18 03:53:37,254 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-18 03:53:37,261 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-18 03:53:37,274 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=409, ProcessCount=172, AvailableMemoryMB=14388
2023-05-18 03:53:37,276 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-18 03:53:37,277 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-05-18 03:53:37,454 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1007055771:84 , srv1026007672:84 , srv1056290599:84 , srv1066067546:84 , srv1089181261:84 , srv1108388525:84 , srv1109530771:84 , srv1126675471:84 , srv1132979602:84 , srv11380650:84 , srv115183137:84 , srv1168928671:84 , srv1191101172:84 , srv1197597871:84 , srv1201185781:84 , srv1206311828:84 , srv1320643974:84 , srv1333161322:84 , srv1379363377:84 , srv1382005830:84 , srv1391856132:84 , srv1411150154:84 , srv1416635458:84 , srv1426302430:84 , srv1428693715:84 , srv1431697818:84 , srv1436325527:84 , srv1459037689:84 , srv1492830753:84 , srv1496893674:84 , srv1519636827:84 , srv1520979472:84 , srv1522661852:84 , srv1570027526:84 , srv159361380:84 , srv160024630:84 , srv1621794040:84 , srv1665373119:84 , srv172449633:84 , srv1735333764:84 , srv1756057685:84 , srv1768879652:84 , srv1828722504:84 , srv1848851127:84 , srv1848866834:84 , srv1872137075:84 , srv1883625081:84 , srv1891891781:84 , srv1908587649:84 , srv1914579305:84 , srv1930265908:84 , srv1947238491:84 , srv1952364360:84 , srv1985321175:84 , srv1986382231:84 , srv2025192941:84 , srv2033061825:84 , srv2056609097:84 , srv206315864:84 , srv2066833334:84 , srv2087527920:84 , srv2093377047:84 , srv2100093742:84 , srv226255825:84 , srv252333017:84 , srv270084948:84 , srv288062859:84 , srv348212258:84 , srv351260249:84 , srv355876627:84 , srv369460026:84 , srv371433994:84 , srv378720436:84 , srv427449443:84 , srv439215223:84 , srv491284116:84 , srv49375231:84 , srv499531280:84 , srv519882145:84 , srv533498267:84 , srv539708364:84 , srv540394685:84 , srv54631820:84 , srv665177022:84 , srv702326996:84 , srv723564468:84 , srv744010009:84 , srv745135611:84 , srv746385734:84 , srv765222084:84 , srv767116200:84 , srv793363591:84 , srv797401641:84 , srv815911109:84 , srv880623112:84 , srv897433548:84 , srv901049437:84 , srv908305730:84 , srv956089948:84 , srv2140612335:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-05-18 03:53:37,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-18 03:53:37,471 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv115183137=10, srv1985321175=53, srv1109530771=6, srv1168928671=11, srv1379363377=18, srv1108388525=5, srv348212258=68, srv1191101172=12, srv765222084=90, srv2056609097=57, srv1768879652=41, srv540394685=82, srv1883625081=46, srv351260249=69, srv539708364=81, srv880623112=95, srv1621794040=36, srv2093377047=61, srv1333161322=17, srv2140612335=63, srv1426302430=23, srv746385734=89, srv1519636827=30, srv371433994=72, srv11380650=9, srv1520979472=31, srv665177022=84, srv1089181261=4, srv744010009=87, srv54631820=83, srv1735333764=39, srv355876627=70, srv2087527920=60, srv2066833334=59, srv1665373119=37, srv1947238491=51, srv1496893674=29, srv378720436=73, srv767116200=91, srv1416635458=22, srv1828722504=42, srv427449443=74, srv270084948=66, srv1007055771=0, srv519882145=79, srv723564468=86, srv1206311828=15, srv1391856132=20, srv908305730=98, srv499531280=78, srv897433548=96, srv745135611=88, srv1522661852=32, srv1914579305=49, srv1756057685=40, srv1848866834=44, srv1459037689=27, srv2025192941=55, srv159361380=34, srv1066067546=3, srv956089948=99, srv1132979602=8, srv160024630=35, srv1201185781=14, srv2100093742=62, srv439215223=75, srv1952364360=52, srv901049437=97, srv2033061825=56, srv491284116=76, srv1197597871=13, srv369460026=71, srv1320643974=16, srv1908587649=48, srv815911109=94, srv1126675471=7, srv1848851127=43, srv172449633=38, srv1986382231=54, srv1872137075=45, srv206315864=58, srv797401641=93, srv793363591=92, srv1411150154=21, srv1382005830=19, srv1026007672=1, srv1056290599=2, srv533498267=80, srv1428693715=24, srv1431697818=25, srv1436325527=26, srv702326996=85, srv1891891781=47, srv1570027526=33, srv1930265908=50, srv1492830753=28, srv226255825=64, srv49375231=77, srv252333017=65, srv288062859=67} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-18 03:53:37,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-18 03:53:37,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-18 03:53:37,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-18 03:53:37,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-18 03:53:37,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-18 03:53:37,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-18 03:53:37,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-18 03:53:37,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-18 03:53:37,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-18 03:53:37,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-18 03:53:37,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-18 03:53:37,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-18 03:53:37,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-18 03:53:37,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-18 03:53:37,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-18 03:53:37,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-18 03:53:37,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-18 03:53:37,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-18 03:53:37,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-18 03:53:37,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-18 03:53:37,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-18 03:53:37,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-18 03:53:37,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-18 03:53:37,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-18 03:53:37,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-18 03:53:37,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-18 03:53:37,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-18 03:53:37,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-18 03:53:37,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-18 03:53:37,545 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-18 03:53:37,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-18 03:53:37,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-18 03:53:37,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-18 03:53:37,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-18 03:53:37,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-18 03:53:37,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-18 03:53:37,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-18 03:53:37,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-18 03:53:37,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-18 03:53:37,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-18 03:53:37,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-18 03:53:37,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-18 03:53:37,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-18 03:53:37,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-18 03:53:37,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-18 03:53:37,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-18 03:53:37,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-18 03:53:37,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-18 03:53:37,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-18 03:53:37,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-18 03:53:37,547 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-18 03:53:37,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-18 03:53:37,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-18 03:53:37,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-18 03:53:37,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-18 03:53:37,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-18 03:53:37,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-18 03:53:37,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-18 03:53:37,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-18 03:53:37,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-18 03:53:37,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-18 03:53:37,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-18 03:53:37,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-18 03:53:37,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-18 03:53:37,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-18 03:53:37,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-18 03:53:37,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-18 03:53:37,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-18 03:53:37,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-18 03:53:37,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-18 03:53:37,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-18 03:53:37,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-18 03:53:37,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-18 03:53:37,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-18 03:53:37,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-18 03:53:37,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-18 03:53:37,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-18 03:53:37,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-18 03:53:37,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-18 03:53:37,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-18 03:53:37,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-18 03:53:37,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-18 03:53:37,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-18 03:53:37,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-18 03:53:37,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-18 03:53:37,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-18 03:53:37,550 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-18 03:53:37,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-18 03:53:37,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-18 03:53:37,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-18 03:53:37,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-18 03:53:37,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-18 03:53:37,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-18 03:53:37,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-18 03:53:37,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-18 03:53:37,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-18 03:53:37,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-18 03:53:37,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-18 03:53:37,551 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-18 03:53:37,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-18 03:53:37,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-18 03:53:37,718 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-18 03:53:37,719 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-18 03:53:37,719 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-18 03:53:38,153 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 581 ms to try 5000 different iterations.  Found a solution that moves 4698 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18105679313683326. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040045179622818755, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.060667813656970916, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.522, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040045179622818755, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17641666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-18 03:53:38,191 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1007055771:83 , srv1056290599:83 , srv1496893674:83 , srv1522661852:83 , srv206315864:83 , srv1108388525:84 , srv11380650:84 , srv1391856132:84 , srv1520979472:84 , srv1621794040:84 , srv1848866834:84 , srv1883625081:84 , srv252333017:84 , srv348212258:84 , srv533498267:84 , srv797401641:84 , srv815911109:84 , srv901049437:84 , srv1168928671:85 , srv1201185781:85 , srv1426302430:85 , srv1436325527:85 , srv1665373119:85 , srv1756057685:85 , srv1828722504:85 , srv1891891781:85 , srv1914579305:85 , srv1986382231:85 , srv2025192941:85 , srv2066833334:85 , srv226255825:85 , srv355876627:85 , srv369460026:85 , srv439215223:85 , srv491284116:85 , srv499531280:85 , srv746385734:85 , srv767116200:85 , srv908305730:85 , srv1026007672:86 , srv1066067546:86 , srv1109530771:86 , srv115183137:86 , srv1197597871:86 , srv1333161322:86 , srv1379363377:86 , srv1411150154:86 , srv1428693715:86 , srv1459037689:86 , srv1492830753:86 , srv1570027526:86 , srv160024630:86 , srv172449633:86 , srv1908587649:86 , srv1952364360:86 , srv2056609097:86 , srv270084948:86 , srv371433994:86 , srv519882145:86 , srv539708364:86 , srv540394685:86 , srv723564468:86 , srv745135611:86 , srv897433548:86 , srv1126675471:87 , srv1191101172:87 , srv1416635458:87 , srv1431697818:87 , srv159361380:87 , srv1985321175:87 , srv351260249:87 , srv427449443:87 , srv665177022:87 , srv744010009:87 , srv765222084:87 , srv793363591:87 , srv1089181261:88 , srv1132979602:88 , srv1206311828:88 , srv1930265908:88 , srv2033061825:88 , srv1519636827:89 , srv1735333764:89 , srv1768879652:89 , srv2100093742:89 , srv378720436:89 , srv702326996:89 , srv1320643974:90 , srv1848851127:90 , srv1947238491:90 , srv2087527920:90 , srv2093377047:90 , srv49375231:90 , srv54631820:90 , srv880623112:90 , srv956089948:90 , srv1382005830:91 , srv1872137075:96 , srv288062859:97 , srv2140612335:446 }
2023-05-18 03:53:38,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-18 03:53:38,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv115183137=10, srv1985321175=53, srv1109530771=6, srv1168928671=11, srv1379363377=18, srv1108388525=5, srv348212258=68, srv1191101172=12, srv765222084=90, srv2056609097=57, srv1768879652=41, srv540394685=82, srv1883625081=46, srv351260249=69, srv539708364=81, srv880623112=95, srv1621794040=36, srv2093377047=61, srv1333161322=17, srv2140612335=63, srv1426302430=23, srv746385734=89, srv1519636827=30, srv371433994=72, srv11380650=9, srv1520979472=31, srv665177022=84, srv1089181261=4, srv744010009=87, srv54631820=83, srv1735333764=39, srv355876627=70, srv2087527920=60, srv2066833334=59, srv1665373119=37, srv1947238491=51, srv1496893674=29, srv378720436=73, srv767116200=91, srv1416635458=22, srv1828722504=42, srv427449443=74, srv270084948=66, srv1007055771=0, srv519882145=79, srv723564468=86, srv1206311828=15, srv1391856132=20, srv908305730=98, srv499531280=78, srv897433548=96, srv745135611=88, srv1522661852=32, srv1914579305=49, srv1756057685=40, srv1848866834=44, srv1459037689=27, srv2025192941=55, srv159361380=34, srv1066067546=3, srv956089948=99, srv1132979602=8, srv160024630=35, srv1201185781=14, srv2100093742=62, srv439215223=75, srv1952364360=52, srv901049437=97, srv2033061825=56, srv491284116=76, srv1197597871=13, srv369460026=71, srv1320643974=16, srv1908587649=48, srv815911109=94, srv1126675471=7, srv1848851127=43, srv172449633=38, srv1986382231=54, srv1872137075=45, srv206315864=58, srv797401641=93, srv793363591=92, srv1411150154=21, srv1382005830=19, srv1026007672=1, srv1056290599=2, srv533498267=80, srv1428693715=24, srv1431697818=25, srv1436325527=26, srv702326996=85, srv1891891781=47, srv1570027526=33, srv1930265908=50, srv1492830753=28, srv226255825=64, srv49375231=77, srv252333017=65, srv288062859=67} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-18 03:53:38,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-18 03:53:38,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-18 03:53:38,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-18 03:53:38,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-18 03:53:38,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-18 03:53:38,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-18 03:53:38,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-18 03:53:38,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-18 03:53:38,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-18 03:53:38,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-18 03:53:38,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-18 03:53:38,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-18 03:53:38,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-18 03:53:38,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-18 03:53:38,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-18 03:53:38,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-18 03:53:38,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-18 03:53:38,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-18 03:53:38,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-18 03:53:38,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-18 03:53:38,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-18 03:53:38,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-18 03:53:38,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-18 03:53:38,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-18 03:53:38,232 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-18 03:53:38,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-18 03:53:38,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-18 03:53:38,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-18 03:53:38,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-18 03:53:38,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-18 03:53:38,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-18 03:53:38,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-18 03:53:38,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-18 03:53:38,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-18 03:53:38,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-18 03:53:38,233 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-18 03:53:38,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-18 03:53:38,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-18 03:53:38,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-18 03:53:38,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-18 03:53:38,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-18 03:53:38,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-18 03:53:38,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-18 03:53:38,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-18 03:53:38,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-18 03:53:38,234 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-18 03:53:38,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-18 03:53:38,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-18 03:53:38,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-18 03:53:38,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-18 03:53:38,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-18 03:53:38,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-18 03:53:38,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-18 03:53:38,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-18 03:53:38,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-18 03:53:38,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-18 03:53:38,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-18 03:53:38,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-18 03:53:38,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-18 03:53:38,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-18 03:53:38,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-18 03:53:38,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-18 03:53:38,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-18 03:53:38,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-18 03:53:38,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-18 03:53:38,236 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-18 03:53:38,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-18 03:53:38,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-18 03:53:38,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-18 03:53:38,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-18 03:53:38,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-18 03:53:38,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-18 03:53:38,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-18 03:53:38,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-18 03:53:38,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-18 03:53:38,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-18 03:53:38,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-18 03:53:38,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-18 03:53:38,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-18 03:53:38,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-18 03:53:38,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-18 03:53:38,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-18 03:53:38,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-18 03:53:38,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-18 03:53:38,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-18 03:53:38,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-18 03:53:38,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-18 03:53:38,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-18 03:53:38,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-18 03:53:38,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-18 03:53:38,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-18 03:53:38,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-18 03:53:38,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-18 03:53:38,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-18 03:53:38,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-18 03:53:38,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-18 03:53:38,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-18 03:53:38,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-18 03:53:38,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-18 03:53:38,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-18 03:53:38,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-18 03:53:38,411 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040045179622818755, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.060667813656970916, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040045179622818755, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17641666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-18 03:53:38,412 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-18 03:53:38,412 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18102389703773084, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040045179622818755, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.060667813656970916, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040045179622818755, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17641666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24141666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-18 03:53:38,822 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 553 ms to try 5000 different iterations.  Found a solution that moves 2647 regions; Going from a computed imbalance of 0.18102389703773084 to a new imbalance of 0.012185302944274489. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023276345351007404); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034657211867887866, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2941111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023276345351007404); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.00875); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04466666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-18 03:53:38,842 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1872137075:75 , srv288062859:76 , srv1382005830:80 , srv54631820:80 , srv1848851127:81 , srv1947238491:82 , srv2087527920:82 , srv2093377047:82 , srv2100093742:82 , srv378720436:82 , srv793363591:82 , srv880623112:82 , srv1519636827:83 , srv1768879652:83 , srv956089948:83 , srv1132979602:84 , srv1191101172:84 , srv1206311828:84 , srv1320643974:84 , srv1416635458:84 , srv1428693715:84 , srv1431697818:84 , srv1570027526:84 , srv1930265908:84 , srv270084948:84 , srv49375231:84 , srv702326996:84 , srv1026007672:85 , srv1066067546:85 , srv1197597871:85 , srv1333161322:85 , srv1379363377:85 , srv1411150154:85 , srv1459037689:85 , srv159361380:85 , srv1735333764:85 , srv1891891781:85 , srv1985321175:85 , srv2033061825:85 , srv351260249:85 , srv519882145:85 , srv665177022:85 , srv745135611:85 , srv765222084:85 , srv1089181261:86 , srv1126675471:86 , srv115183137:86 , srv1168928671:86 , srv1426302430:86 , srv1621794040:86 , srv1756057685:86 , srv1883625081:86 , srv1952364360:86 , srv2056609097:86 , srv369460026:86 , srv371433994:86 , srv427449443:86 , srv439215223:86 , srv499531280:86 , srv723564468:86 , srv746385734:86 , srv767116200:86 , srv815911109:86 , srv897433548:86 , srv901049437:86 , srv908305730:86 , srv1108388525:87 , srv1109530771:87 , srv1436325527:87 , srv1496893674:87 , srv160024630:87 , srv1665373119:87 , srv172449633:87 , srv1908587649:87 , srv252333017:87 , srv533498267:87 , srv539708364:87 , srv744010009:87 , srv1056290599:88 , srv1492830753:88 , srv1522661852:88 , srv1828722504:88 , srv1914579305:88 , srv2025192941:88 , srv2066833334:88 , srv226255825:88 , srv348212258:88 , srv355876627:88 , srv491284116:88 , srv797401641:88 , srv11380650:89 , srv1391856132:89 , srv1520979472:89 , srv1848866834:89 , srv1986382231:89 , srv206315864:89 , srv540394685:89 , srv1007055771:90 , srv1201185781:92 , srv2140612335:535 }
2023-05-18 03:53:38,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-18 03:53:38,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv115183137=10, srv1985321175=53, srv1109530771=6, srv1168928671=11, srv1379363377=18, srv1108388525=5, srv348212258=68, srv1191101172=12, srv765222084=90, srv2056609097=57, srv1768879652=41, srv540394685=82, srv1883625081=46, srv351260249=69, srv539708364=81, srv880623112=95, srv1621794040=36, srv2093377047=61, srv1333161322=17, srv2140612335=63, srv1426302430=23, srv746385734=89, srv1519636827=30, srv371433994=72, srv11380650=9, srv1520979472=31, srv665177022=84, srv1089181261=4, srv744010009=87, srv54631820=83, srv1735333764=39, srv355876627=70, srv2087527920=60, srv2066833334=59, srv1665373119=37, srv1947238491=51, srv1496893674=29, srv378720436=73, srv767116200=91, srv1416635458=22, srv1828722504=42, srv427449443=74, srv270084948=66, srv1007055771=0, srv519882145=79, srv723564468=86, srv1206311828=15, srv1391856132=20, srv908305730=98, srv499531280=78, srv897433548=96, srv745135611=88, srv1522661852=32, srv1914579305=49, srv1756057685=40, srv1848866834=44, srv1459037689=27, srv2025192941=55, srv159361380=34, srv1066067546=3, srv956089948=99, srv1132979602=8, srv160024630=35, srv1201185781=14, srv2100093742=62, srv439215223=75, srv1952364360=52, srv901049437=97, srv2033061825=56, srv491284116=76, srv1197597871=13, srv369460026=71, srv1320643974=16, srv1908587649=48, srv815911109=94, srv1126675471=7, srv1848851127=43, srv172449633=38, srv1986382231=54, srv1872137075=45, srv206315864=58, srv797401641=93, srv793363591=92, srv1411150154=21, srv1382005830=19, srv1026007672=1, srv1056290599=2, srv533498267=80, srv1428693715=24, srv1431697818=25, srv1436325527=26, srv702326996=85, srv1891891781=47, srv1570027526=33, srv1930265908=50, srv1492830753=28, srv226255825=64, srv49375231=77, srv252333017=65, srv288062859=67} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-18 03:53:38,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-18 03:53:38,869 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-18 03:53:38,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-18 03:53:38,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-18 03:53:38,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-18 03:53:38,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-18 03:53:38,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-18 03:53:38,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-18 03:53:38,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-18 03:53:38,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-18 03:53:38,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-18 03:53:38,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-18 03:53:38,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-18 03:53:38,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-18 03:53:38,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-18 03:53:38,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-18 03:53:38,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-18 03:53:38,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-18 03:53:38,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-18 03:53:38,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-18 03:53:38,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-18 03:53:38,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-18 03:53:38,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-18 03:53:38,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-18 03:53:38,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-18 03:53:38,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-18 03:53:38,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-18 03:53:38,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-18 03:53:38,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-18 03:53:38,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-18 03:53:38,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-18 03:53:38,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-18 03:53:38,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-18 03:53:38,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-18 03:53:38,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-18 03:53:38,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-18 03:53:38,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-18 03:53:38,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-18 03:53:38,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-18 03:53:38,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-18 03:53:38,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-18 03:53:38,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-18 03:53:38,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-18 03:53:38,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-18 03:53:38,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-18 03:53:38,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-18 03:53:38,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-18 03:53:38,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-18 03:53:38,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-18 03:53:38,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-18 03:53:38,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-18 03:53:38,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-18 03:53:38,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-18 03:53:38,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-18 03:53:38,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-18 03:53:38,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-18 03:53:38,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-18 03:53:38,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-18 03:53:38,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-18 03:53:38,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-18 03:53:38,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-18 03:53:38,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-18 03:53:38,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-18 03:53:38,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-18 03:53:38,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-18 03:53:38,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-18 03:53:38,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-18 03:53:38,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-18 03:53:38,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-18 03:53:38,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-18 03:53:38,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-18 03:53:38,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-18 03:53:38,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-18 03:53:38,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-18 03:53:38,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-18 03:53:38,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-18 03:53:38,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-18 03:53:38,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-18 03:53:38,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-18 03:53:38,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-18 03:53:38,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-18 03:53:38,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-18 03:53:38,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-18 03:53:38,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-18 03:53:38,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-18 03:53:38,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-18 03:53:38,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-18 03:53:38,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-18 03:53:38,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-18 03:53:38,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-18 03:53:38,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-18 03:53:38,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-18 03:53:38,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-18 03:53:38,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-18 03:53:38,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-18 03:53:38,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-18 03:53:38,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-18 03:53:38,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-18 03:53:38,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-18 03:53:38,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-18 03:53:38,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-18 03:53:39,058 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023276345351007404); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034657211867887866, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023276345351007404); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.00875); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04466666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-18 03:53:39,059 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-18 03:53:39,059 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.012166768254124613, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023276345351007404); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034657211867887866, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023276345351007404); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.00875); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04466666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-18 03:53:39,385 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 431 ms to try 5000 different iterations.  Found a solution that moves 977 regions; Going from a computed imbalance of 0.012166768254124613 to a new imbalance of 7.492630562360825E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006011586269238993); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0068329432425405096); MoveCostFunction : (multiplier=7.0, imbalance=0.10855555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006011586269238993); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-18 03:53:39,387 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1201185781:80 , srv1735333764:83 , srv1986382231:83 , srv49375231:83 , srv540394685:83 , srv744010009:83 , srv956089948:83 , srv1089181261:84 , srv1320643974:84 , srv1382005830:84 , srv1492830753:84 , srv1520979472:84 , srv159361380:84 , srv160024630:84 , srv1828722504:84 , srv1908587649:84 , srv1914579305:84 , srv2025192941:84 , srv2066833334:84 , srv288062859:84 , srv355876627:84 , srv702326996:84 , srv1126675471:85 , srv11380650:85 , srv115183137:85 , srv1519636827:85 , srv1665373119:85 , srv1768879652:85 , srv1872137075:85 , srv1930265908:85 , srv1947238491:85 , srv1952364360:85 , srv1985321175:85 , srv2056609097:85 , srv226255825:85 , srv351260249:85 , srv427449443:85 , srv491284116:85 , srv723564468:85 , srv765222084:85 , srv797401641:85 , srv1007055771:86 , srv1026007672:86 , srv1066067546:86 , srv1109530771:86 , srv1132979602:86 , srv1168928671:86 , srv1206311828:86 , srv1391856132:86 , srv1411150154:86 , srv1426302430:86 , srv1431697818:86 , srv1436325527:86 , srv1570027526:86 , srv172449633:86 , srv1756057685:86 , srv1848851127:86 , srv1848866834:86 , srv1891891781:86 , srv2033061825:86 , srv206315864:86 , srv2087527920:86 , srv2093377047:86 , srv252333017:86 , srv348212258:86 , srv369460026:86 , srv519882145:86 , srv533498267:86 , srv539708364:86 , srv54631820:86 , srv665177022:86 , srv767116200:86 , srv880623112:86 , srv897433548:86 , srv908305730:86 , srv1056290599:87 , srv1108388525:87 , srv1197597871:87 , srv1333161322:87 , srv1379363377:87 , srv1416635458:87 , srv1459037689:87 , srv1496893674:87 , srv1522661852:87 , srv1883625081:87 , srv2100093742:87 , srv270084948:87 , srv371433994:87 , srv378720436:87 , srv439215223:87 , srv499531280:87 , srv745135611:87 , srv746385734:87 , srv815911109:87 , srv1191101172:88 , srv1428693715:88 , srv1621794040:88 , srv793363591:88 , srv901049437:88 , srv2140612335:530 }
2023-05-18 03:53:39,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-18 03:53:39,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv115183137=10, srv1985321175=53, srv1109530771=6, srv1168928671=11, srv1379363377=18, srv1108388525=5, srv348212258=68, srv1191101172=12, srv765222084=90, srv2056609097=57, srv1768879652=41, srv540394685=82, srv1883625081=46, srv351260249=69, srv539708364=81, srv880623112=95, srv1621794040=36, srv2093377047=61, srv1333161322=17, srv2140612335=63, srv1426302430=23, srv746385734=89, srv1519636827=30, srv371433994=72, srv11380650=9, srv1520979472=31, srv665177022=84, srv1089181261=4, srv744010009=87, srv54631820=83, srv1735333764=39, srv355876627=70, srv2087527920=60, srv2066833334=59, srv1665373119=37, srv1947238491=51, srv1496893674=29, srv378720436=73, srv767116200=91, srv1416635458=22, srv1828722504=42, srv427449443=74, srv270084948=66, srv1007055771=0, srv519882145=79, srv723564468=86, srv1206311828=15, srv1391856132=20, srv908305730=98, srv499531280=78, srv897433548=96, srv745135611=88, srv1522661852=32, srv1914579305=49, srv1756057685=40, srv1848866834=44, srv1459037689=27, srv2025192941=55, srv159361380=34, srv1066067546=3, srv956089948=99, srv1132979602=8, srv160024630=35, srv1201185781=14, srv2100093742=62, srv439215223=75, srv1952364360=52, srv901049437=97, srv2033061825=56, srv491284116=76, srv1197597871=13, srv369460026=71, srv1320643974=16, srv1908587649=48, srv815911109=94, srv1126675471=7, srv1848851127=43, srv172449633=38, srv1986382231=54, srv1872137075=45, srv206315864=58, srv797401641=93, srv793363591=92, srv1411150154=21, srv1382005830=19, srv1026007672=1, srv1056290599=2, srv533498267=80, srv1428693715=24, srv1431697818=25, srv1436325527=26, srv702326996=85, srv1891891781=47, srv1570027526=33, srv1930265908=50, srv1492830753=28, srv226255825=64, srv49375231=77, srv252333017=65, srv288062859=67} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-18 03:53:39,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-18 03:53:39,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-18 03:53:39,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-18 03:53:39,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-18 03:53:39,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-18 03:53:39,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-18 03:53:39,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-18 03:53:39,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-18 03:53:39,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-18 03:53:39,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-18 03:53:39,412 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-18 03:53:39,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-18 03:53:39,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-18 03:53:39,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-18 03:53:39,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-18 03:53:39,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-18 03:53:39,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-18 03:53:39,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-18 03:53:39,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-18 03:53:39,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-18 03:53:39,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-18 03:53:39,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-18 03:53:39,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-18 03:53:39,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-18 03:53:39,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-18 03:53:39,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-18 03:53:39,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-18 03:53:39,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-18 03:53:39,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-18 03:53:39,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-18 03:53:39,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-18 03:53:39,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-18 03:53:39,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-18 03:53:39,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-18 03:53:39,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-18 03:53:39,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-18 03:53:39,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-18 03:53:39,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-18 03:53:39,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-18 03:53:39,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-18 03:53:39,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-18 03:53:39,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-18 03:53:39,414 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-18 03:53:39,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-18 03:53:39,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-18 03:53:39,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-18 03:53:39,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-18 03:53:39,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-18 03:53:39,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-18 03:53:39,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-18 03:53:39,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-18 03:53:39,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-18 03:53:39,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-18 03:53:39,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-18 03:53:39,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-18 03:53:39,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-18 03:53:39,416 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-18 03:53:39,417 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-18 03:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-18 03:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-18 03:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-18 03:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-18 03:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-18 03:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-18 03:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-18 03:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-18 03:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-18 03:53:39,418 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-18 03:53:39,526 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006011586269238993); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0068329432425405096); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006011586269238993); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-18 03:53:39,526 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-05-18 03:53:39,527 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.424219560183157E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006011586269238993); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0068329432425405096); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006011586269238993); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00425); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-05-18 03:53:39,846 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 413 ms to try 5000 different iterations.  Found a solution that moves 272 regions; Going from a computed imbalance of 7.424219560183157E-4 to a new imbalance of 2.1647264206539006E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.030222222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-18 03:53:39,847 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1007055771:83 , srv1109530771:83 , srv1191101172:83 , srv1391856132:83 , srv172449633:83 , srv1848866834:83 , srv2033061825:83 , srv371433994:83 , srv539708364:83 , srv1056290599:84 , srv1089181261:84 , srv1108388525:84 , srv1126675471:84 , srv1132979602:84 , srv11380650:84 , srv1197597871:84 , srv1206311828:84 , srv1320643974:84 , srv1333161322:84 , srv1379363377:84 , srv1416635458:84 , srv1426302430:84 , srv1428693715:84 , srv1436325527:84 , srv1459037689:84 , srv1492830753:84 , srv1520979472:84 , srv1621794040:84 , srv1883625081:84 , srv1947238491:84 , srv206315864:84 , srv2087527920:84 , srv2093377047:84 , srv2100093742:84 , srv226255825:84 , srv348212258:84 , srv355876627:84 , srv378720436:84 , srv427449443:84 , srv439215223:84 , srv491284116:84 , srv499531280:84 , srv540394685:84 , srv665177022:84 , srv746385734:84 , srv880623112:84 , srv897433548:84 , srv901049437:84 , srv1026007672:85 , srv1066067546:85 , srv115183137:85 , srv1168928671:85 , srv1201185781:85 , srv1431697818:85 , srv1496893674:85 , srv1519636827:85 , srv1522661852:85 , srv160024630:85 , srv1665373119:85 , srv1735333764:85 , srv1756057685:85 , srv1768879652:85 , srv1848851127:85 , srv1908587649:85 , srv1914579305:85 , srv1930265908:85 , srv1952364360:85 , srv1985321175:85 , srv1986382231:85 , srv2025192941:85 , srv2056609097:85 , srv2066833334:85 , srv252333017:85 , srv270084948:85 , srv288062859:85 , srv351260249:85 , srv369460026:85 , srv519882145:85 , srv533498267:85 , srv54631820:85 , srv702326996:85 , srv723564468:85 , srv744010009:85 , srv745135611:85 , srv765222084:85 , srv767116200:85 , srv797401641:85 , srv815911109:85 , srv908305730:85 , srv1382005830:86 , srv1411150154:86 , srv1570027526:86 , srv159361380:86 , srv1828722504:86 , srv1891891781:86 , srv49375231:86 , srv793363591:86 , srv956089948:86 , srv1872137075:87 , srv2140612335:631 }
2023-05-18 03:53:39,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-18 03:53:39,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv115183137=10, srv1985321175=53, srv1109530771=6, srv1168928671=11, srv1379363377=18, srv1108388525=5, srv348212258=68, srv1191101172=12, srv765222084=90, srv2056609097=57, srv1768879652=41, srv540394685=82, srv1883625081=46, srv351260249=69, srv539708364=81, srv880623112=95, srv1621794040=36, srv2093377047=61, srv1333161322=17, srv2140612335=63, srv1426302430=23, srv746385734=89, srv1519636827=30, srv371433994=72, srv11380650=9, srv1520979472=31, srv665177022=84, srv1089181261=4, srv744010009=87, srv54631820=83, srv1735333764=39, srv355876627=70, srv2087527920=60, srv2066833334=59, srv1665373119=37, srv1947238491=51, srv1496893674=29, srv378720436=73, srv767116200=91, srv1416635458=22, srv1828722504=42, srv427449443=74, srv270084948=66, srv1007055771=0, srv519882145=79, srv723564468=86, srv1206311828=15, srv1391856132=20, srv908305730=98, srv499531280=78, srv897433548=96, srv745135611=88, srv1522661852=32, srv1914579305=49, srv1756057685=40, srv1848866834=44, srv1459037689=27, srv2025192941=55, srv159361380=34, srv1066067546=3, srv956089948=99, srv1132979602=8, srv160024630=35, srv1201185781=14, srv2100093742=62, srv439215223=75, srv1952364360=52, srv901049437=97, srv2033061825=56, srv491284116=76, srv1197597871=13, srv369460026=71, srv1320643974=16, srv1908587649=48, srv815911109=94, srv1126675471=7, srv1848851127=43, srv172449633=38, srv1986382231=54, srv1872137075=45, srv206315864=58, srv797401641=93, srv793363591=92, srv1411150154=21, srv1382005830=19, srv1026007672=1, srv1056290599=2, srv533498267=80, srv1428693715=24, srv1431697818=25, srv1436325527=26, srv702326996=85, srv1891891781=47, srv1570027526=33, srv1930265908=50, srv1492830753=28, srv226255825=64, srv49375231=77, srv252333017=65, srv288062859=67} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-18 03:53:39,870 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-18 03:53:39,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-18 03:53:39,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-18 03:53:39,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-18 03:53:39,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-18 03:53:39,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-18 03:53:39,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-18 03:53:39,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-18 03:53:39,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-18 03:53:39,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-18 03:53:39,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-18 03:53:39,871 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-18 03:53:39,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-18 03:53:39,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-18 03:53:39,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-18 03:53:39,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-18 03:53:39,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-18 03:53:39,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-18 03:53:39,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-18 03:53:39,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-18 03:53:39,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-18 03:53:39,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-18 03:53:39,872 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-18 03:53:39,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-18 03:53:39,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-18 03:53:39,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-18 03:53:39,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-18 03:53:39,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-18 03:53:39,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-18 03:53:39,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-18 03:53:39,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-05-18 03:53:39,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-05-18 03:53:39,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-05-18 03:53:39,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-05-18 03:53:39,873 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-05-18 03:53:39,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-05-18 03:53:39,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-05-18 03:53:39,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-05-18 03:53:39,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-05-18 03:53:39,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-05-18 03:53:39,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-05-18 03:53:39,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-05-18 03:53:39,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-05-18 03:53:39,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-05-18 03:53:39,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-05-18 03:53:39,874 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-05-18 03:53:39,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-05-18 03:53:39,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-05-18 03:53:39,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-05-18 03:53:39,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-05-18 03:53:39,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-05-18 03:53:39,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-05-18 03:53:39,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-05-18 03:53:39,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-05-18 03:53:39,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-05-18 03:53:39,875 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-05-18 03:53:39,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-05-18 03:53:39,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-05-18 03:53:39,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-05-18 03:53:39,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-05-18 03:53:39,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-05-18 03:53:39,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-05-18 03:53:39,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-05-18 03:53:39,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-05-18 03:53:39,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-05-18 03:53:39,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-05-18 03:53:39,876 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-05-18 03:53:39,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-05-18 03:53:39,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-05-18 03:53:39,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-05-18 03:53:39,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-05-18 03:53:39,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-05-18 03:53:39,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-05-18 03:53:39,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-05-18 03:53:39,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-05-18 03:53:39,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-05-18 03:53:39,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-05-18 03:53:39,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-05-18 03:53:39,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-05-18 03:53:39,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-05-18 03:53:39,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-05-18 03:53:39,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-05-18 03:53:39,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-05-18 03:53:39,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-05-18 03:53:39,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-05-18 03:53:39,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-05-18 03:53:39,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-05-18 03:53:39,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-05-18 03:53:39,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-05-18 03:53:39,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-05-18 03:53:39,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-05-18 03:53:39,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-05-18 03:53:39,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-05-18 03:53:39,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-05-18 03:53:39,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-05-18 03:53:39,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-05-18 03:53:39,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-05-18 03:53:39,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-05-18 03:53:39,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-05-18 03:53:39,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-05-18 03:53:39,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-05-18 03:53:39,981 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.9742679499033804E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-18 03:53:39,982 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1007055771:83 , srv1109530771:83 , srv1191101172:83 , srv1391856132:83 , srv172449633:83 , srv1848866834:83 , srv2033061825:83 , srv371433994:83 , srv539708364:83 , srv1056290599:84 , srv1089181261:84 , srv1108388525:84 , srv1126675471:84 , srv1132979602:84 , srv11380650:84 , srv1197597871:84 , srv1206311828:84 , srv1320643974:84 , srv1333161322:84 , srv1379363377:84 , srv1416635458:84 , srv1426302430:84 , srv1428693715:84 , srv1436325527:84 , srv1459037689:84 , srv1492830753:84 , srv1520979472:84 , srv1621794040:84 , srv1883625081:84 , srv1947238491:84 , srv206315864:84 , srv2087527920:84 , srv2093377047:84 , srv2100093742:84 , srv226255825:84 , srv348212258:84 , srv355876627:84 , srv378720436:84 , srv427449443:84 , srv439215223:84 , srv491284116:84 , srv499531280:84 , srv540394685:84 , srv665177022:84 , srv746385734:84 , srv880623112:84 , srv897433548:84 , srv901049437:84 , srv1026007672:85 , srv1066067546:85 , srv115183137:85 , srv1168928671:85 , srv1201185781:85 , srv1431697818:85 , srv1496893674:85 , srv1519636827:85 , srv1522661852:85 , srv160024630:85 , srv1665373119:85 , srv1735333764:85 , srv1756057685:85 , srv1768879652:85 , srv1848851127:85 , srv1908587649:85 , srv1914579305:85 , srv1930265908:85 , srv1952364360:85 , srv1985321175:85 , srv1986382231:85 , srv2025192941:85 , srv2056609097:85 , srv2066833334:85 , srv252333017:85 , srv270084948:85 , srv288062859:85 , srv351260249:85 , srv369460026:85 , srv519882145:85 , srv533498267:85 , srv54631820:85 , srv702326996:85 , srv723564468:85 , srv744010009:85 , srv745135611:85 , srv765222084:85 , srv767116200:85 , srv797401641:85 , srv815911109:85 , srv908305730:85 , srv1382005830:86 , srv1411150154:86 , srv1570027526:86 , srv159361380:86 , srv1828722504:86 , srv1891891781:86 , srv49375231:86 , srv793363591:86 , srv956089948:86 , srv1872137075:87 , srv2140612335:631 }
2023-05-18 03:53:40,021 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=409 (was 409), ProcessCount=172 (was 172), AvailableMemoryMB=13891 (was 14388)
2023-05-18 03:53:40,031 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=264, MaxFileDescriptor=60000, SystemLoadAverage=409, ProcessCount=172, AvailableMemoryMB=13890
2023-05-18 03:53:40,032 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-05-18 03:53:40,032 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-05-18 03:53:40,045 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1095637745:84 , srv1191318314:84 , srv1244484585:84 , srv133788691:84 , srv1379223280:84 , srv1420001978:84 , srv1481465365:84 , srv1550588166:84 , srv1560964278:84 , srv1626700433:84 , srv1732666692:84 , srv1777263657:84 , srv1830102185:84 , srv1872380197:84 , srv1941176397:84 , srv194870012:84 , srv202475342:84 , srv2035161328:84 , srv2137880597:84 , srv23664262:84 , srv256969005:84 , srv305438646:84 , srv41785564:84 , srv424676853:84 , srv644919108:84 , srv651647816:84 , srv678714308:84 , srv722989665:84 , srv99963405:84 , srv874382460:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-05-18 03:53:40,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-18 03:53:40,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1420001978=5, srv722989665=27, srv651647816=25, srv305438646=21, srv1481465365=6, srv1560964278=8, srv1872380197=13, srv874382460=28, srv202475342=16, srv256969005=20, srv1550588166=7, srv644919108=24, srv1777263657=11, srv2137880597=18, srv1626700433=9, srv1830102185=12, srv41785564=22, srv1244484585=2, srv1941176397=14, srv2035161328=17, srv133788691=3, srv424676853=23, srv99963405=29, srv1095637745=0, srv678714308=26, srv1379223280=4, srv1191318314=1, srv194870012=15, srv23664262=19, srv1732666692=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-18 03:53:40,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-18 03:53:40,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-18 03:53:40,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-18 03:53:40,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-18 03:53:40,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-18 03:53:40,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-18 03:53:40,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-18 03:53:40,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-18 03:53:40,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-18 03:53:40,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-18 03:53:40,056 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-18 03:53:40,057 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-18 03:53:40,073 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0732328937414295, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-18 03:53:40,073 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912423797721186, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0732328937414295, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-05-18 03:55:40,061 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6005304 different iterations.  Found a solution that moves 1852 regions; Going from a computed imbalance of 0.9912423797721186 to a new imbalance of 4.322660390073086E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6859259259259259, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-18 03:55:40,068 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1095637745:90 , srv1191318314:90 , srv1244484585:90 , srv133788691:90 , srv1379223280:90 , srv1420001978:90 , srv1481465365:90 , srv1550588166:90 , srv1560964278:90 , srv1626700433:90 , srv1732666692:90 , srv1777263657:90 , srv1830102185:90 , srv1872380197:90 , srv1941176397:90 , srv194870012:90 , srv202475342:90 , srv2035161328:90 , srv2137880597:90 , srv23664262:90 , srv256969005:90 , srv305438646:90 , srv41785564:90 , srv424676853:90 , srv644919108:90 , srv651647816:90 , srv678714308:90 , srv722989665:90 , srv874382460:90 , srv99963405:90 }
2023-05-18 03:55:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-05-18 03:55:40,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1420001978=5, srv722989665=27, srv651647816=25, srv305438646=21, srv1481465365=6, srv1560964278=8, srv1872380197=13, srv874382460=28, srv202475342=16, srv256969005=20, srv1550588166=7, srv644919108=24, srv1777263657=11, srv2137880597=18, srv1626700433=9, srv1830102185=12, srv41785564=22, srv1244484585=2, srv1941176397=14, srv2035161328=17, srv133788691=3, srv424676853=23, srv99963405=29, srv1095637745=0, srv678714308=26, srv1379223280=4, srv1191318314=1, srv194870012=15, srv23664262=19, srv1732666692=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-05-18 03:55:40,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-05-18 03:55:40,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-05-18 03:55:40,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-05-18 03:55:40,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-05-18 03:55:40,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-05-18 03:55:40,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-05-18 03:55:40,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-05-18 03:55:40,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-05-18 03:55:40,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-05-18 03:55:40,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-05-18 03:55:40,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-05-18 03:55:40,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-05-18 03:55:40,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-05-18 03:55:40,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-05-18 03:55:40,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-05-18 03:55:40,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-05-18 03:55:40,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-05-18 03:55:40,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-05-18 03:55:40,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-05-18 03:55:40,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-05-18 03:55:40,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-05-18 03:55:40,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-05-18 03:55:40,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-05-18 03:55:40,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-05-18 03:55:40,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-05-18 03:55:40,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-05-18 03:55:40,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-05-18 03:55:40,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-05-18 03:55:40,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-05-18 03:55:40,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-05-18 03:55:40,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-05-18 03:55:40,093 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-05-18 03:55:40,094 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1095637745:90 , srv1191318314:90 , srv1244484585:90 , srv133788691:90 , srv1379223280:90 , srv1420001978:90 , srv1481465365:90 , srv1550588166:90 , srv1560964278:90 , srv1626700433:90 , srv1732666692:90 , srv1777263657:90 , srv1830102185:90 , srv1872380197:90 , srv1941176397:90 , srv194870012:90 , srv202475342:90 , srv2035161328:90 , srv2137880597:90 , srv23664262:90 , srv256969005:90 , srv305438646:90 , srv41785564:90 , srv424676853:90 , srv644919108:90 , srv651647816:90 , srv678714308:90 , srv722989665:90 , srv874382460:90 , srv99963405:90 }
2023-05-18 03:55:40,111 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=264 (was 264), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=190 (was 409), ProcessCount=169 (was 172), AvailableMemoryMB=13653 (was 13890)