2023-03-30 11:56:31,233 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-30 11:56:31,558 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-30 11:56:31,665 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-30 11:56:31,729 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-30 11:56:31,736 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-30 11:56:31,748 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=326, ProcessCount=171, AvailableMemoryMB=5390
2023-03-30 11:56:31,750 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-30 11:56:31,751 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-30 11:56:31,871 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv100712771:84 , srv1026005681:84 , srv1032053282:84 , srv1049982541:84 , srv1058385826:84 , srv1070108661:84 , srv1070382894:84 , srv1112515456:84 , srv1155049324:84 , srv1254691391:84 , srv128545803:84 , srv1288838092:84 , srv1313540447:84 , srv1318184835:84 , srv1342879800:84 , srv1352239484:84 , srv1357276823:84 , srv1377954916:84 , srv1420717421:84 , srv1456423145:84 , srv1459659045:84 , srv1484850586:84 , srv1574396785:84 , srv1577110202:84 , srv1618929227:84 , srv169193547:84 , srv172847294:84 , srv1786469585:84 , srv1794466557:84 , srv1841001947:84 , srv1860040335:84 , srv1888949732:84 , srv1891040909:84 , srv1895542587:84 , srv1902199191:84 , srv1919685301:84 , srv1930240588:84 , srv1951380770:84 , srv1960543275:84 , srv1970553488:84 , srv1984675963:84 , srv1993287376:84 , srv1999798532:84 , srv2002353855:84 , srv20169943:84 , srv2061702373:84 , srv2067431747:84 , srv2074558358:84 , srv2086447125:84 , srv211132874:84 , srv2114875425:84 , srv2138824388:84 , srv2145188114:84 , srv224732323:84 , srv241119089:84 , srv244384521:84 , srv247696493:84 , srv252760389:84 , srv276362676:84 , srv293010684:84 , srv293144571:84 , srv331632304:84 , srv337767872:84 , srv386183647:84 , srv400657499:84 , srv402917751:84 , srv43044303:84 , srv4369351:84 , srv456720514:84 , srv477255435:84 , srv482575142:84 , srv491241573:84 , srv508913414:84 , srv509851558:84 , srv517855368:84 , srv528731401:84 , srv562682495:84 , srv581976262:84 , srv604935839:84 , srv614501203:84 , srv625344613:84 , srv656046458:84 , srv659590682:84 , srv67984277:84 , srv693804902:84 , srv699996922:84 , srv709743923:84 , srv716163996:84 , srv724930708:84 , srv747945639:84 , srv79456217:84 , srv840232947:84 , srv862241766:84 , srv879433544:84 , srv884127230:84 , srv89787509:84 , srv925979956:84 , srv928264569:84 , srv958529886:84 , srv2094491527:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-30 11:56:31,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-30 11:56:31,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1288838092=11, srv1456423145=19, srv1112515456=7, srv2086447125=48, srv293144571=61, srv1984675963=40, srv1577110202=23, srv456720514=69, srv562682495=77, srv1049982541=3, srv67984277=84, srv169193547=25, srv1841001947=29, srv509851558=74, srv656046458=82, srv1254691391=9, srv276362676=59, srv331632304=62, srv20169943=44, srv1070382894=6, srv1891040909=32, srv1919685301=35, srv293010684=60, srv2074558358=47, srv337767872=63, srv241119089=55, srv928264569=98, srv1342879800=14, srv724930708=89, srv1999798532=42, srv4369351=68, srv1574396785=22, srv1357276823=16, srv879433544=94, srv79456217=91, srv1155049324=8, srv747945639=90, srv1070108661=5, srv100712771=0, srv1459659045=20, srv699996922=86, srv1484850586=21, srv482575142=71, srv1895542587=33, srv1902199191=34, srv1026005681=1, srv1794466557=28, srv1960543275=38, srv43044303=67, srv89787509=96, srv211132874=50, srv247696493=57, srv1618929227=24, srv581976262=78, srv1377954916=17, srv1951380770=37, srv2061702373=45, srv604935839=79, srv709743923=87, srv172847294=26, srv244384521=56, srv477255435=70, srv1970553488=39, srv402917751=66, srv1032053282=2, srv491241573=72, srv1318184835=13, srv2114875425=51, srv659590682=83, srv2138824388=52, srv716163996=88, srv224732323=54, srv625344613=81, srv1352239484=15, srv1860040335=30, srv1058385826=4, srv1786469585=27, srv862241766=93, srv128545803=10, srv1888949732=31, srv2067431747=46, srv2145188114=53, srv884127230=95, srv925979956=97, srv693804902=85, srv252760389=58, srv508913414=73, srv1313540447=12, srv2094491527=49, srv1993287376=41, srv2002353855=43, srv386183647=64, srv528731401=76, srv400657499=65, srv517855368=75, srv1420717421=18, srv840232947=92, srv1930240588=36, srv614501203=80, srv958529886=99} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-30 11:56:31,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-30 11:56:31,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-30 11:56:31,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-30 11:56:31,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-30 11:56:31,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-30 11:56:31,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-30 11:56:31,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-30 11:56:31,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-30 11:56:31,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-30 11:56:31,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-30 11:56:31,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-30 11:56:31,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-30 11:56:31,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-30 11:56:31,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-30 11:56:31,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-30 11:56:31,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-30 11:56:31,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-30 11:56:31,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-30 11:56:31,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-30 11:56:31,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-30 11:56:31,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-30 11:56:31,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-30 11:56:31,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-30 11:56:31,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-30 11:56:31,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-30 11:56:31,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-30 11:56:31,973 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-30 11:56:31,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-30 11:56:31,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-30 11:56:31,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-30 11:56:31,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-30 11:56:31,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-30 11:56:31,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-30 11:56:31,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-30 11:56:31,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-30 11:56:31,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-30 11:56:31,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-30 11:56:31,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-30 11:56:31,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-30 11:56:31,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-30 11:56:31,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-30 11:56:31,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-30 11:56:31,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-30 11:56:31,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-30 11:56:31,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-30 11:56:31,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-30 11:56:31,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-30 11:56:31,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-30 11:56:31,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-30 11:56:31,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-30 11:56:31,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-30 11:56:31,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-30 11:56:31,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-30 11:56:31,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-30 11:56:31,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-30 11:56:31,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-30 11:56:31,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-30 11:56:31,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-30 11:56:31,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-30 11:56:31,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-30 11:56:31,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-30 11:56:31,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-30 11:56:31,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-30 11:56:31,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-30 11:56:31,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-30 11:56:31,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-30 11:56:31,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-30 11:56:31,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-30 11:56:31,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-30 11:56:31,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-30 11:56:31,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-30 11:56:31,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-30 11:56:31,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-30 11:56:31,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-30 11:56:31,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-30 11:56:31,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-30 11:56:31,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-30 11:56:31,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-30 11:56:31,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-30 11:56:31,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-30 11:56:31,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-30 11:56:31,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-30 11:56:31,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-30 11:56:31,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-30 11:56:31,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-30 11:56:31,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-30 11:56:31,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-30 11:56:31,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-30 11:56:31,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-30 11:56:31,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-30 11:56:31,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-30 11:56:31,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-30 11:56:31,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-30 11:56:31,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-30 11:56:31,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-30 11:56:31,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-30 11:56:31,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-30 11:56:31,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-30 11:56:31,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-30 11:56:31,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-30 11:56:31,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-30 11:56:32,158 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-30 11:56:32,158 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-30 11:56:32,158 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-30 11:56:32,803 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 801 ms to try 5000 different iterations.  Found a solution that moves 4629 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18648263245523. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03862101101191541, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05600424867641028, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5143333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03862101101191541, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18208333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24533333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-30 11:56:32,827 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1841001947:80 , srv276362676:81 , srv1070382894:82 , srv386183647:82 , srv43044303:82 , srv1919685301:83 , srv1999798532:83 , srv247696493:83 , srv293144571:83 , srv4369351:83 , srv614501203:83 , srv747945639:83 , srv840232947:83 , srv879433544:83 , srv1032053282:84 , srv1070108661:84 , srv1155049324:84 , srv1342879800:84 , srv1459659045:84 , srv1970553488:84 , srv477255435:84 , srv699996922:84 , srv1112515456:85 , srv1288838092:85 , srv1313540447:85 , srv1618929227:85 , srv1930240588:85 , srv252760389:85 , srv517855368:85 , srv528731401:85 , srv625344613:85 , srv656046458:85 , srv693804902:85 , srv716163996:85 , srv862241766:85 , srv884127230:85 , srv925979956:85 , srv100712771:86 , srv1357276823:86 , srv1420717421:86 , srv1574396785:86 , srv211132874:86 , srv2138824388:86 , srv293010684:86 , srv456720514:86 , srv482575142:86 , srv604935839:86 , srv79456217:86 , srv89787509:86 , srv1254691391:87 , srv1318184835:87 , srv1377954916:87 , srv1456423145:87 , srv1484850586:87 , srv172847294:87 , srv1786469585:87 , srv1860040335:87 , srv1888949732:87 , srv1891040909:87 , srv1902199191:87 , srv1993287376:87 , srv20169943:87 , srv2074558358:87 , srv2086447125:87 , srv241119089:87 , srv244384521:87 , srv337767872:87 , srv402917751:87 , srv491241573:87 , srv509851558:87 , srv958529886:87 , srv1049982541:88 , srv1058385826:88 , srv1352239484:88 , srv169193547:88 , srv2067431747:88 , srv724930708:88 , srv1026005681:89 , srv2002353855:89 , srv2114875425:89 , srv2145188114:89 , srv224732323:89 , srv331632304:89 , srv400657499:89 , srv562682495:89 , srv659590682:89 , srv1895542587:90 , srv1984675963:90 , srv581976262:90 , srv709743923:90 , srv2061702373:91 , srv508913414:91 , srv1794466557:92 , srv1960543275:92 , srv67984277:92 , srv928264569:92 , srv1577110202:93 , srv1951380770:93 , srv128545803:95 , srv2094491527:433 }
2023-03-30 11:56:32,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-30 11:56:32,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1288838092=11, srv1456423145=19, srv1112515456=7, srv2086447125=48, srv293144571=61, srv1984675963=40, srv1577110202=23, srv456720514=69, srv562682495=77, srv1049982541=3, srv67984277=84, srv169193547=25, srv1841001947=29, srv509851558=74, srv656046458=82, srv1254691391=9, srv276362676=59, srv331632304=62, srv20169943=44, srv1070382894=6, srv1891040909=32, srv1919685301=35, srv293010684=60, srv2074558358=47, srv337767872=63, srv241119089=55, srv928264569=98, srv1342879800=14, srv724930708=89, srv1999798532=42, srv4369351=68, srv1574396785=22, srv1357276823=16, srv879433544=94, srv79456217=91, srv1155049324=8, srv747945639=90, srv1070108661=5, srv100712771=0, srv1459659045=20, srv699996922=86, srv1484850586=21, srv482575142=71, srv1895542587=33, srv1902199191=34, srv1026005681=1, srv1794466557=28, srv1960543275=38, srv43044303=67, srv89787509=96, srv211132874=50, srv247696493=57, srv1618929227=24, srv581976262=78, srv1377954916=17, srv1951380770=37, srv2061702373=45, srv604935839=79, srv709743923=87, srv172847294=26, srv244384521=56, srv477255435=70, srv1970553488=39, srv402917751=66, srv1032053282=2, srv491241573=72, srv1318184835=13, srv2114875425=51, srv659590682=83, srv2138824388=52, srv716163996=88, srv224732323=54, srv625344613=81, srv1352239484=15, srv1860040335=30, srv1058385826=4, srv1786469585=27, srv862241766=93, srv128545803=10, srv1888949732=31, srv2067431747=46, srv2145188114=53, srv884127230=95, srv925979956=97, srv693804902=85, srv252760389=58, srv508913414=73, srv1313540447=12, srv2094491527=49, srv1993287376=41, srv2002353855=43, srv386183647=64, srv528731401=76, srv400657499=65, srv517855368=75, srv1420717421=18, srv840232947=92, srv1930240588=36, srv614501203=80, srv958529886=99} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-30 11:56:32,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-30 11:56:32,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-30 11:56:32,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-30 11:56:32,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-30 11:56:32,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-30 11:56:32,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-30 11:56:32,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-30 11:56:32,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-30 11:56:32,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-30 11:56:32,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-30 11:56:32,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-30 11:56:32,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-30 11:56:32,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-30 11:56:32,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-30 11:56:32,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-30 11:56:32,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-30 11:56:32,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-30 11:56:32,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-30 11:56:32,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-30 11:56:32,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-30 11:56:32,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-30 11:56:32,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-30 11:56:32,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-30 11:56:32,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-30 11:56:32,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-30 11:56:32,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-30 11:56:32,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-30 11:56:32,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-30 11:56:32,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-30 11:56:32,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-30 11:56:32,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-30 11:56:32,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-30 11:56:32,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-30 11:56:32,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-30 11:56:32,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-30 11:56:32,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-30 11:56:32,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-30 11:56:32,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-30 11:56:32,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-30 11:56:32,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-30 11:56:32,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-30 11:56:32,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-30 11:56:32,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-30 11:56:32,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-30 11:56:32,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-30 11:56:32,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-30 11:56:32,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-30 11:56:32,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-30 11:56:32,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-30 11:56:32,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-30 11:56:32,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-30 11:56:32,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-30 11:56:32,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-30 11:56:32,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-30 11:56:32,854 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-30 11:56:32,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-30 11:56:32,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-30 11:56:32,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-30 11:56:32,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-30 11:56:32,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-30 11:56:32,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-30 11:56:32,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-30 11:56:32,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-30 11:56:32,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-30 11:56:32,855 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-30 11:56:32,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-30 11:56:32,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-30 11:56:32,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-30 11:56:32,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-30 11:56:32,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-30 11:56:32,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-30 11:56:32,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-30 11:56:32,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-30 11:56:32,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-30 11:56:32,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-30 11:56:32,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-30 11:56:32,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-30 11:56:32,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-30 11:56:32,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-30 11:56:32,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-30 11:56:32,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-30 11:56:32,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-30 11:56:32,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-30 11:56:32,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-30 11:56:32,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-30 11:56:32,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-30 11:56:32,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-30 11:56:32,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-30 11:56:32,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-30 11:56:32,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-30 11:56:32,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-30 11:56:32,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-30 11:56:32,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-30 11:56:32,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-30 11:56:32,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-30 11:56:32,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-30 11:56:32,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-30 11:56:32,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-30 11:56:32,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-30 11:56:32,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-30 11:56:32,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-30 11:56:33,038 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03862101101191541, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05600424867641028, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03862101101191541, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18208333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24533333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-30 11:56:33,038 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-30 11:56:33,038 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1864502195044541, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03862101101191541, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05600424867641028, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03862101101191541, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18208333333333335, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24533333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-30 11:56:33,320 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 438 ms to try 5000 different iterations.  Found a solution that moves 2672 regions; Going from a computed imbalance of 0.1864502195044541 to a new imbalance of 0.01593743841428384. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020737132923346314); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034220516814387404, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29688888888888887, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020737132923346314); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0465, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-30 11:56:33,329 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv128545803:77 , srv1951380770:77 , srv928264569:78 , srv1577110202:79 , srv508913414:79 , srv1794466557:80 , srv1960543275:80 , srv67984277:80 , srv2002353855:81 , srv2061702373:81 , srv2145188114:81 , srv1026005681:82 , srv1895542587:82 , srv2114875425:82 , srv491241573:82 , srv709743923:82 , srv1574396785:83 , srv1786469585:83 , srv1993287376:83 , srv400657499:83 , srv724930708:83 , srv1058385826:84 , srv1318184835:84 , srv1357276823:84 , srv1377954916:84 , srv169193547:84 , srv172847294:84 , srv1888949732:84 , srv241119089:84 , srv331632304:84 , srv581976262:84 , srv716163996:84 , srv958529886:84 , srv1049982541:85 , srv1254691391:85 , srv1484850586:85 , srv1891040909:85 , srv2067431747:85 , srv244384521:85 , srv509851558:85 , srv528731401:85 , srv562682495:85 , srv659590682:85 , srv100712771:86 , srv1032053282:86 , srv1352239484:86 , srv1902199191:86 , srv1930240588:86 , srv2074558358:86 , srv2138824388:86 , srv224732323:86 , srv252760389:86 , srv402917751:86 , srv482575142:86 , srv517855368:86 , srv625344613:86 , srv747945639:86 , srv79456217:86 , srv862241766:86 , srv884127230:86 , srv89787509:86 , srv1288838092:87 , srv1313540447:87 , srv1860040335:87 , srv1970553488:87 , srv2086447125:87 , srv293010684:87 , srv925979956:87 , srv1112515456:88 , srv1420717421:88 , srv1456423145:88 , srv1459659045:88 , srv1618929227:88 , srv20169943:88 , srv247696493:88 , srv337767872:88 , srv456720514:88 , srv477255435:88 , srv656046458:88 , srv1070108661:89 , srv1155049324:89 , srv1342879800:89 , srv1919685301:89 , srv1999798532:89 , srv211132874:89 , srv293144571:89 , srv386183647:89 , srv614501203:89 , srv693804902:89 , srv840232947:89 , srv1070382894:90 , srv1984675963:90 , srv276362676:90 , srv43044303:90 , srv4369351:90 , srv699996922:90 , srv604935839:91 , srv879433544:91 , srv1841001947:93 , srv2094491527:525 }
2023-03-30 11:56:33,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-30 11:56:33,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1288838092=11, srv1456423145=19, srv1112515456=7, srv2086447125=48, srv293144571=61, srv1984675963=40, srv1577110202=23, srv456720514=69, srv562682495=77, srv1049982541=3, srv67984277=84, srv169193547=25, srv1841001947=29, srv509851558=74, srv656046458=82, srv1254691391=9, srv276362676=59, srv331632304=62, srv20169943=44, srv1070382894=6, srv1891040909=32, srv1919685301=35, srv293010684=60, srv2074558358=47, srv337767872=63, srv241119089=55, srv928264569=98, srv1342879800=14, srv724930708=89, srv1999798532=42, srv4369351=68, srv1574396785=22, srv1357276823=16, srv879433544=94, srv79456217=91, srv1155049324=8, srv747945639=90, srv1070108661=5, srv100712771=0, srv1459659045=20, srv699996922=86, srv1484850586=21, srv482575142=71, srv1895542587=33, srv1902199191=34, srv1026005681=1, srv1794466557=28, srv1960543275=38, srv43044303=67, srv89787509=96, srv211132874=50, srv247696493=57, srv1618929227=24, srv581976262=78, srv1377954916=17, srv1951380770=37, srv2061702373=45, srv604935839=79, srv709743923=87, srv172847294=26, srv244384521=56, srv477255435=70, srv1970553488=39, srv402917751=66, srv1032053282=2, srv491241573=72, srv1318184835=13, srv2114875425=51, srv659590682=83, srv2138824388=52, srv716163996=88, srv224732323=54, srv625344613=81, srv1352239484=15, srv1860040335=30, srv1058385826=4, srv1786469585=27, srv862241766=93, srv128545803=10, srv1888949732=31, srv2067431747=46, srv2145188114=53, srv884127230=95, srv925979956=97, srv693804902=85, srv252760389=58, srv508913414=73, srv1313540447=12, srv2094491527=49, srv1993287376=41, srv2002353855=43, srv386183647=64, srv528731401=76, srv400657499=65, srv517855368=75, srv1420717421=18, srv840232947=92, srv1930240588=36, srv614501203=80, srv958529886=99} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-30 11:56:33,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-30 11:56:33,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-30 11:56:33,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-30 11:56:33,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-30 11:56:33,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-30 11:56:33,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-30 11:56:33,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-30 11:56:33,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-30 11:56:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-30 11:56:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-30 11:56:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-30 11:56:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-30 11:56:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-30 11:56:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-30 11:56:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-30 11:56:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-30 11:56:33,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-30 11:56:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-30 11:56:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-30 11:56:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-30 11:56:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-30 11:56:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-30 11:56:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-30 11:56:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-30 11:56:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-30 11:56:33,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-30 11:56:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-30 11:56:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-30 11:56:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-30 11:56:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-30 11:56:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-30 11:56:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-30 11:56:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-30 11:56:33,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-30 11:56:33,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-30 11:56:33,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-30 11:56:33,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-30 11:56:33,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-30 11:56:33,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-30 11:56:33,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-30 11:56:33,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-30 11:56:33,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-30 11:56:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-30 11:56:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-30 11:56:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-30 11:56:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-30 11:56:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-30 11:56:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-30 11:56:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-30 11:56:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-30 11:56:33,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-30 11:56:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-30 11:56:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-30 11:56:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-30 11:56:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-30 11:56:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-30 11:56:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-30 11:56:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-30 11:56:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-30 11:56:33,371 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-30 11:56:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-30 11:56:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-30 11:56:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-30 11:56:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-30 11:56:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-30 11:56:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-30 11:56:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-30 11:56:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-30 11:56:33,372 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-30 11:56:33,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-30 11:56:33,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-30 11:56:33,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-30 11:56:33,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-30 11:56:33,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-30 11:56:33,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-30 11:56:33,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-30 11:56:33,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-30 11:56:33,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-30 11:56:33,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-30 11:56:33,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-30 11:56:33,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-30 11:56:33,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-30 11:56:33,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-30 11:56:33,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-30 11:56:33,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-30 11:56:33,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-30 11:56:33,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-30 11:56:33,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-30 11:56:33,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-30 11:56:33,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-30 11:56:33,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-30 11:56:33,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-30 11:56:33,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-30 11:56:33,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-30 11:56:33,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-30 11:56:33,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-30 11:56:33,375 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-30 11:56:33,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-30 11:56:33,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-30 11:56:33,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-30 11:56:33,376 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-30 11:56:33,493 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020737132923346314); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034220516814387404, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020737132923346314); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0465, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-30 11:56:33,493 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-30 11:56:33,493 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015918728670392467, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020737132923346314); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.034220516814387404, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020737132923346314); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01275); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0465, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-30 11:56:33,714 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 321 ms to try 5000 different iterations.  Found a solution that moves 1039 regions; Going from a computed imbalance of 0.015918728670392467 to a new imbalance of 6.577409738412256E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054295474061417486); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007027283689263065); MoveCostFunction : (multiplier=7.0, imbalance=0.11544444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054295474061417486); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-30 11:56:33,717 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1984675963:78 , srv1456423145:82 , srv20169943:82 , srv211132874:82 , srv337767872:82 , srv604935839:82 , srv1352239484:83 , srv1420717421:83 , srv224732323:83 , srv699996922:83 , srv879433544:83 , srv1070108661:84 , srv1112515456:84 , srv1342879800:84 , srv1618929227:84 , srv1841001947:84 , srv1860040335:84 , srv2086447125:84 , srv293010684:84 , srv402917751:84 , srv4369351:84 , srv456720514:84 , srv581976262:84 , srv656046458:84 , srv659590682:84 , srv1049982541:85 , srv1070382894:85 , srv1155049324:85 , srv1254691391:85 , srv128545803:85 , srv1288838092:85 , srv1313540447:85 , srv1459659045:85 , srv169193547:85 , srv1891040909:85 , srv1895542587:85 , srv1902199191:85 , srv2067431747:85 , srv2138824388:85 , srv244384521:85 , srv331632304:85 , srv477255435:85 , srv482575142:85 , srv562682495:85 , srv693804902:85 , srv925979956:85 , srv100712771:86 , srv1026005681:86 , srv1058385826:86 , srv1377954916:86 , srv1484850586:86 , srv1577110202:86 , srv172847294:86 , srv1794466557:86 , srv1919685301:86 , srv1930240588:86 , srv1960543275:86 , srv1970553488:86 , srv1999798532:86 , srv2061702373:86 , srv2074558358:86 , srv2114875425:86 , srv241119089:86 , srv247696493:86 , srv252760389:86 , srv276362676:86 , srv386183647:86 , srv400657499:86 , srv43044303:86 , srv509851558:86 , srv614501203:86 , srv625344613:86 , srv67984277:86 , srv709743923:86 , srv79456217:86 , srv840232947:86 , srv862241766:86 , srv884127230:86 , srv89787509:86 , srv1032053282:87 , srv1318184835:87 , srv1786469585:87 , srv1888949732:87 , srv1951380770:87 , srv2145188114:87 , srv293144571:87 , srv508913414:87 , srv517855368:87 , srv528731401:87 , srv724930708:87 , srv747945639:87 , srv958529886:87 , srv1357276823:88 , srv1993287376:88 , srv2002353855:88 , srv928264569:88 , srv1574396785:89 , srv491241573:89 , srv716163996:89 , srv2094491527:548 }
2023-03-30 11:56:33,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-30 11:56:33,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1288838092=11, srv1456423145=19, srv1112515456=7, srv2086447125=48, srv293144571=61, srv1984675963=40, srv1577110202=23, srv456720514=69, srv562682495=77, srv1049982541=3, srv67984277=84, srv169193547=25, srv1841001947=29, srv509851558=74, srv656046458=82, srv1254691391=9, srv276362676=59, srv331632304=62, srv20169943=44, srv1070382894=6, srv1891040909=32, srv1919685301=35, srv293010684=60, srv2074558358=47, srv337767872=63, srv241119089=55, srv928264569=98, srv1342879800=14, srv724930708=89, srv1999798532=42, srv4369351=68, srv1574396785=22, srv1357276823=16, srv879433544=94, srv79456217=91, srv1155049324=8, srv747945639=90, srv1070108661=5, srv100712771=0, srv1459659045=20, srv699996922=86, srv1484850586=21, srv482575142=71, srv1895542587=33, srv1902199191=34, srv1026005681=1, srv1794466557=28, srv1960543275=38, srv43044303=67, srv89787509=96, srv211132874=50, srv247696493=57, srv1618929227=24, srv581976262=78, srv1377954916=17, srv1951380770=37, srv2061702373=45, srv604935839=79, srv709743923=87, srv172847294=26, srv244384521=56, srv477255435=70, srv1970553488=39, srv402917751=66, srv1032053282=2, srv491241573=72, srv1318184835=13, srv2114875425=51, srv659590682=83, srv2138824388=52, srv716163996=88, srv224732323=54, srv625344613=81, srv1352239484=15, srv1860040335=30, srv1058385826=4, srv1786469585=27, srv862241766=93, srv128545803=10, srv1888949732=31, srv2067431747=46, srv2145188114=53, srv884127230=95, srv925979956=97, srv693804902=85, srv252760389=58, srv508913414=73, srv1313540447=12, srv2094491527=49, srv1993287376=41, srv2002353855=43, srv386183647=64, srv528731401=76, srv400657499=65, srv517855368=75, srv1420717421=18, srv840232947=92, srv1930240588=36, srv614501203=80, srv958529886=99} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-30 11:56:33,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-30 11:56:33,731 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-30 11:56:33,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-30 11:56:33,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-30 11:56:33,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-30 11:56:33,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-30 11:56:33,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-30 11:56:33,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-30 11:56:33,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-30 11:56:33,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-30 11:56:33,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-30 11:56:33,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-30 11:56:33,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-30 11:56:33,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-30 11:56:33,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-30 11:56:33,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-30 11:56:33,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-30 11:56:33,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-30 11:56:33,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-30 11:56:33,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-30 11:56:33,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-30 11:56:33,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-30 11:56:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-30 11:56:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-30 11:56:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-30 11:56:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-30 11:56:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-30 11:56:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-30 11:56:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-30 11:56:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-30 11:56:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-30 11:56:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-30 11:56:33,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-30 11:56:33,844 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054295474061417486); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007027283689263065); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054295474061417486); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-30 11:56:33,844 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-30 11:56:33,844 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.504657403445421E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0054295474061417486); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007027283689263065); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0054295474061417486); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.6666666666666666E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-30 11:56:34,155 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 409 ms to try 5000 different iterations.  Found a solution that moves 249 regions; Going from a computed imbalance of 6.504657403445421E-4 to a new imbalance of 2.005843425854682E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.027666666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-30 11:56:34,156 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv562682495:82 , srv100712771:83 , srv2074558358:83 , srv293144571:83 , srv400657499:83 , srv604935839:83 , srv693804902:83 , srv724930708:83 , srv1026005681:84 , srv1049982541:84 , srv1058385826:84 , srv1112515456:84 , srv1155049324:84 , srv1318184835:84 , srv1357276823:84 , srv1484850586:84 , srv1574396785:84 , srv1794466557:84 , srv1860040335:84 , srv1895542587:84 , srv1902199191:84 , srv1919685301:84 , srv1930240588:84 , srv1960543275:84 , srv1993287376:84 , srv1999798532:84 , srv2002353855:84 , srv2061702373:84 , srv2086447125:84 , srv224732323:84 , srv247696493:84 , srv331632304:84 , srv43044303:84 , srv456720514:84 , srv491241573:84 , srv509851558:84 , srv517855368:84 , srv581976262:84 , srv614501203:84 , srv659590682:84 , srv67984277:84 , srv709743923:84 , srv79456217:84 , srv840232947:84 , srv928264569:84 , srv958529886:84 , srv1032053282:85 , srv1070108661:85 , srv1070382894:85 , srv1254691391:85 , srv128545803:85 , srv1288838092:85 , srv1313540447:85 , srv1342879800:85 , srv1352239484:85 , srv1377954916:85 , srv1456423145:85 , srv1459659045:85 , srv1577110202:85 , srv1618929227:85 , srv169193547:85 , srv172847294:85 , srv1786469585:85 , srv1841001947:85 , srv1888949732:85 , srv1891040909:85 , srv1951380770:85 , srv1970553488:85 , srv1984675963:85 , srv20169943:85 , srv2067431747:85 , srv211132874:85 , srv2114875425:85 , srv2138824388:85 , srv2145188114:85 , srv241119089:85 , srv252760389:85 , srv276362676:85 , srv293010684:85 , srv337767872:85 , srv402917751:85 , srv4369351:85 , srv477255435:85 , srv482575142:85 , srv508913414:85 , srv528731401:85 , srv625344613:85 , srv656046458:85 , srv699996922:85 , srv716163996:85 , srv862241766:85 , srv879433544:85 , srv884127230:85 , srv89787509:85 , srv925979956:85 , srv1420717421:86 , srv244384521:86 , srv386183647:86 , srv747945639:86 , srv2094491527:636 }
2023-03-30 11:56:34,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-30 11:56:34,159 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1288838092=11, srv1456423145=19, srv1112515456=7, srv2086447125=48, srv293144571=61, srv1984675963=40, srv1577110202=23, srv456720514=69, srv562682495=77, srv1049982541=3, srv67984277=84, srv169193547=25, srv1841001947=29, srv509851558=74, srv656046458=82, srv1254691391=9, srv276362676=59, srv331632304=62, srv20169943=44, srv1070382894=6, srv1891040909=32, srv1919685301=35, srv293010684=60, srv2074558358=47, srv337767872=63, srv241119089=55, srv928264569=98, srv1342879800=14, srv724930708=89, srv1999798532=42, srv4369351=68, srv1574396785=22, srv1357276823=16, srv879433544=94, srv79456217=91, srv1155049324=8, srv747945639=90, srv1070108661=5, srv100712771=0, srv1459659045=20, srv699996922=86, srv1484850586=21, srv482575142=71, srv1895542587=33, srv1902199191=34, srv1026005681=1, srv1794466557=28, srv1960543275=38, srv43044303=67, srv89787509=96, srv211132874=50, srv247696493=57, srv1618929227=24, srv581976262=78, srv1377954916=17, srv1951380770=37, srv2061702373=45, srv604935839=79, srv709743923=87, srv172847294=26, srv244384521=56, srv477255435=70, srv1970553488=39, srv402917751=66, srv1032053282=2, srv491241573=72, srv1318184835=13, srv2114875425=51, srv659590682=83, srv2138824388=52, srv716163996=88, srv224732323=54, srv625344613=81, srv1352239484=15, srv1860040335=30, srv1058385826=4, srv1786469585=27, srv862241766=93, srv128545803=10, srv1888949732=31, srv2067431747=46, srv2145188114=53, srv884127230=95, srv925979956=97, srv693804902=85, srv252760389=58, srv508913414=73, srv1313540447=12, srv2094491527=49, srv1993287376=41, srv2002353855=43, srv386183647=64, srv528731401=76, srv400657499=65, srv517855368=75, srv1420717421=18, srv840232947=92, srv1930240588=36, srv614501203=80, srv958529886=99} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-30 11:56:34,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-30 11:56:34,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-30 11:56:34,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-30 11:56:34,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-30 11:56:34,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-30 11:56:34,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-30 11:56:34,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-30 11:56:34,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-30 11:56:34,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-30 11:56:34,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-30 11:56:34,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-30 11:56:34,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-30 11:56:34,174 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-30 11:56:34,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-30 11:56:34,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-30 11:56:34,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-30 11:56:34,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-30 11:56:34,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-30 11:56:34,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-30 11:56:34,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-30 11:56:34,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-30 11:56:34,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-30 11:56:34,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-30 11:56:34,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-30 11:56:34,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-30 11:56:34,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-30 11:56:34,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-30 11:56:34,175 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-30 11:56:34,285 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8314898993220366E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-30 11:56:34,285 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv562682495:82 , srv100712771:83 , srv2074558358:83 , srv293144571:83 , srv400657499:83 , srv604935839:83 , srv693804902:83 , srv724930708:83 , srv1026005681:84 , srv1049982541:84 , srv1058385826:84 , srv1112515456:84 , srv1155049324:84 , srv1318184835:84 , srv1357276823:84 , srv1484850586:84 , srv1574396785:84 , srv1794466557:84 , srv1860040335:84 , srv1895542587:84 , srv1902199191:84 , srv1919685301:84 , srv1930240588:84 , srv1960543275:84 , srv1993287376:84 , srv1999798532:84 , srv2002353855:84 , srv2061702373:84 , srv2086447125:84 , srv224732323:84 , srv247696493:84 , srv331632304:84 , srv43044303:84 , srv456720514:84 , srv491241573:84 , srv509851558:84 , srv517855368:84 , srv581976262:84 , srv614501203:84 , srv659590682:84 , srv67984277:84 , srv709743923:84 , srv79456217:84 , srv840232947:84 , srv928264569:84 , srv958529886:84 , srv1032053282:85 , srv1070108661:85 , srv1070382894:85 , srv1254691391:85 , srv128545803:85 , srv1288838092:85 , srv1313540447:85 , srv1342879800:85 , srv1352239484:85 , srv1377954916:85 , srv1456423145:85 , srv1459659045:85 , srv1577110202:85 , srv1618929227:85 , srv169193547:85 , srv172847294:85 , srv1786469585:85 , srv1841001947:85 , srv1888949732:85 , srv1891040909:85 , srv1951380770:85 , srv1970553488:85 , srv1984675963:85 , srv20169943:85 , srv2067431747:85 , srv211132874:85 , srv2114875425:85 , srv2138824388:85 , srv2145188114:85 , srv241119089:85 , srv252760389:85 , srv276362676:85 , srv293010684:85 , srv337767872:85 , srv402917751:85 , srv4369351:85 , srv477255435:85 , srv482575142:85 , srv508913414:85 , srv528731401:85 , srv625344613:85 , srv656046458:85 , srv699996922:85 , srv716163996:85 , srv862241766:85 , srv879433544:85 , srv884127230:85 , srv89787509:85 , srv925979956:85 , srv1420717421:86 , srv244384521:86 , srv386183647:86 , srv747945639:86 , srv2094491527:636 }
2023-03-30 11:56:34,314 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=326 (was 326), ProcessCount=171 (was 171), AvailableMemoryMB=4897 (was 5390)
2023-03-30 11:56:34,324 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=326, ProcessCount=171, AvailableMemoryMB=4896
2023-03-30 11:56:34,325 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-30 11:56:34,325 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-30 11:56:34,334 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1129837716:84 , srv120095549:84 , srv1209915049:84 , srv1322306257:84 , srv1435817336:84 , srv1502062745:84 , srv1534103413:84 , srv1574735627:84 , srv1608006473:84 , srv162893486:84 , srv1776318410:84 , srv1792533669:84 , srv1894242720:84 , srv2055171023:84 , srv2067387387:84 , srv2073084648:84 , srv2096592493:84 , srv291897666:84 , srv295474488:84 , srv318734977:84 , srv320462833:84 , srv347810775:84 , srv587751309:84 , srv645600956:84 , srv65753204:84 , srv776737725:84 , srv875567154:84 , srv963652481:84 , srv987352592:84 , srv714149346:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-30 11:56:34,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-30 11:56:34,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1209915049=2, srv2055171023=13, srv1534103413=6, srv1435817336=4, srv2067387387=14, srv291897666=17, srv347810775=21, srv320462833=20, srv963652481=28, srv587751309=22, srv776737725=26, srv1776318410=10, srv875567154=27, srv1322306257=3, srv318734977=19, srv1129837716=0, srv987352592=29, srv120095549=1, srv1608006473=8, srv714149346=25, srv1894242720=12, srv1792533669=11, srv1502062745=5, srv162893486=9, srv645600956=23, srv65753204=24, srv1574735627=7, srv2073084648=15, srv2096592493=16, srv295474488=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-30 11:56:34,343 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-30 11:56:34,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-30 11:56:34,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-30 11:56:34,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-30 11:56:34,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-30 11:56:34,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-30 11:56:34,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-30 11:56:34,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-30 11:56:34,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-30 11:56:34,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-30 11:56:34,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-30 11:56:34,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-30 11:56:34,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-30 11:56:34,344 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-30 11:56:34,360 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1092308752821183, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-30 11:56:34,360 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912537226185578, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1092308752821183, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-30 11:58:34,347 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6889621 different iterations.  Found a solution that moves 1854 regions; Going from a computed imbalance of 0.9912537226185578 to a new imbalance of 4.3273284898463826E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-30 11:58:34,354 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1129837716:90 , srv120095549:90 , srv1209915049:90 , srv1322306257:90 , srv1435817336:90 , srv1502062745:90 , srv1534103413:90 , srv1574735627:90 , srv1608006473:90 , srv162893486:90 , srv1776318410:90 , srv1792533669:90 , srv1894242720:90 , srv2055171023:90 , srv2067387387:90 , srv2073084648:90 , srv2096592493:90 , srv291897666:90 , srv295474488:90 , srv318734977:90 , srv320462833:90 , srv347810775:90 , srv587751309:90 , srv645600956:90 , srv65753204:90 , srv714149346:90 , srv776737725:90 , srv875567154:90 , srv963652481:90 , srv987352592:90 }
2023-03-30 11:58:34,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-30 11:58:34,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1209915049=2, srv2055171023=13, srv1534103413=6, srv1435817336=4, srv2067387387=14, srv291897666=17, srv347810775=21, srv320462833=20, srv963652481=28, srv587751309=22, srv776737725=26, srv1776318410=10, srv875567154=27, srv1322306257=3, srv318734977=19, srv1129837716=0, srv987352592=29, srv120095549=1, srv1608006473=8, srv714149346=25, srv1894242720=12, srv1792533669=11, srv1502062745=5, srv162893486=9, srv645600956=23, srv65753204=24, srv1574735627=7, srv2073084648=15, srv2096592493=16, srv295474488=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-30 11:58:34,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-30 11:58:34,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-30 11:58:34,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-30 11:58:34,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-30 11:58:34,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-30 11:58:34,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-30 11:58:34,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-30 11:58:34,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-30 11:58:34,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-30 11:58:34,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-30 11:58:34,372 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-30 11:58:34,372 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1129837716:90 , srv120095549:90 , srv1209915049:90 , srv1322306257:90 , srv1435817336:90 , srv1502062745:90 , srv1534103413:90 , srv1574735627:90 , srv1608006473:90 , srv162893486:90 , srv1776318410:90 , srv1792533669:90 , srv1894242720:90 , srv2055171023:90 , srv2067387387:90 , srv2073084648:90 , srv2096592493:90 , srv291897666:90 , srv295474488:90 , srv318734977:90 , srv320462833:90 , srv347810775:90 , srv587751309:90 , srv645600956:90 , srv65753204:90 , srv714149346:90 , srv776737725:90 , srv875567154:90 , srv963652481:90 , srv987352592:90 }
2023-03-30 11:58:34,384 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=134 (was 326), ProcessCount=169 (was 171), AvailableMemoryMB=4810 (was 4896)