2023-02-11 03:55:59,159 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-02-11 03:55:59,469 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-02-11 03:55:59,585 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-02-11 03:55:59,642 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-02-11 03:55:59,649 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-02-11 03:55:59,662 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=109, ProcessCount=168, AvailableMemoryMB=3765
2023-02-11 03:55:59,664 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-02-11 03:55:59,665 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-02-11 03:55:59,779 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1005540877:84 , srv1011008458:84 , srv1025220556:84 , srv1078738101:84 , srv1150378456:84 , srv1191672231:84 , srv1200224599:84 , srv1205914559:84 , srv1216909676:84 , srv1230299602:84 , srv1265461773:84 , srv1273302204:84 , srv1277996084:84 , srv1365046903:84 , srv1412563751:84 , srv1429830901:84 , srv1461996381:84 , srv1479574219:84 , srv1485775565:84 , srv15395759:84 , srv1558186916:84 , srv157937333:84 , srv1581920051:84 , srv1610487780:84 , srv1613624140:84 , srv1625865534:84 , srv1654068194:84 , srv1671308341:84 , srv1696939577:84 , srv1698375738:84 , srv1726730302:84 , srv1730306016:84 , srv1748327787:84 , srv1752795630:84 , srv177848805:84 , srv1786947528:84 , srv1787702660:84 , srv1798220426:84 , srv1803609856:84 , srv1861314718:84 , srv1875377077:84 , srv1904459476:84 , srv1958375183:84 , srv1967885778:84 , srv1974297545:84 , srv2001254925:84 , srv20356281:84 , srv2048017920:84 , srv205362884:84 , srv2054891350:84 , srv2072048533:84 , srv2103969816:84 , srv228592358:84 , srv239924548:84 , srv277165825:84 , srv277606235:84 , srv278163448:84 , srv28970393:84 , srv299668954:84 , srv312383356:84 , srv318205916:84 , srv33231229:84 , srv335553573:84 , srv341462400:84 , srv342706440:84 , srv356727189:84 , srv450407410:84 , srv482838438:84 , srv483602120:84 , srv515900047:84 , srv550616423:84 , srv553042203:84 , srv56323706:84 , srv576105633:84 , srv601589319:84 , srv635201423:84 , srv641764602:84 , srv658908385:84 , srv681726610:84 , srv700811291:84 , srv7026039:84 , srv705085369:84 , srv753435890:84 , srv768651621:84 , srv770284184:84 , srv787304599:84 , srv791767047:84 , srv810948958:84 , srv812452801:84 , srv824264789:84 , srv848966791:84 , srv874130901:84 , srv874220531:84 , srv894338769:84 , srv91265282:84 , srv92217382:84 , srv981258659:84 , srv985912693:84 , srv997221324:84 , srv250882279:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-02-11 03:55:59,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-02-11 03:55:59,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1787702660=36, srv1011008458=1, srv1875377077=40, srv1461996381=16, srv1967885778=43, srv312383356=60, srv810948958=88, srv1798220426=37, srv1752795630=33, srv812452801=89, srv177848805=34, srv1696939577=28, srv1479574219=17, srv894338769=94, srv450407410=67, srv2103969816=51, srv1265461773=10, srv299668954=59, srv981258659=97, srv2054891350=49, srv483602120=69, srv1786947528=35, srv56323706=73, srv1005540877=0, srv2001254925=45, srv515900047=70, srv874130901=92, srv1412563751=14, srv848966791=91, srv91265282=95, srv1974297545=44, srv342706440=65, srv15395759=19, srv277606235=56, srv239924548=53, srv2048017920=47, srv335553573=63, srv157937333=21, srv1191672231=5, srv550616423=71, srv1205914559=7, srv1671308341=27, srv635201423=76, srv705085369=82, srv985912693=98, srv482838438=68, srv681726610=79, srv874220531=93, srv753435890=83, srv1216909676=8, srv1958375183=42, srv824264789=90, srv1625865534=25, srv700811291=80, srv553042203=72, srv770284184=85, srv28970393=58, srv641764602=77, srv1610487780=23, srv1273302204=11, srv997221324=99, srv601589319=75, srv1698375738=29, srv341462400=64, srv1200224599=6, srv1613624140=24, srv277165825=55, srv1277996084=12, srv1654068194=26, srv1726730302=30, srv318205916=61, srv33231229=62, srv1429830901=15, srv787304599=86, srv356727189=66, srv1803609856=38, srv7026039=81, srv658908385=78, srv1748327787=32, srv1581920051=22, srv1730306016=31, srv1150378456=4, srv228592358=52, srv1078738101=3, srv278163448=57, srv92217382=96, srv205362884=48, srv20356281=46, srv791767047=87, srv576105633=74, srv1861314718=39, srv1485775565=18, srv1025220556=2, srv1558186916=20, srv768651621=84, srv1904459476=41, srv2072048533=50, srv1365046903=13, srv250882279=54, srv1230299602=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-02-11 03:55:59,877 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-02-11 03:55:59,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-02-11 03:55:59,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-02-11 03:55:59,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-02-11 03:55:59,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-02-11 03:55:59,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-02-11 03:55:59,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-02-11 03:55:59,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-02-11 03:55:59,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-02-11 03:55:59,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-02-11 03:55:59,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-02-11 03:55:59,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-02-11 03:55:59,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-02-11 03:55:59,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-02-11 03:55:59,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-02-11 03:55:59,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-02-11 03:55:59,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-02-11 03:55:59,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-02-11 03:55:59,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-02-11 03:55:59,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-02-11 03:55:59,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-02-11 03:55:59,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-02-11 03:55:59,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-02-11 03:55:59,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-02-11 03:55:59,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-02-11 03:55:59,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-02-11 03:55:59,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-02-11 03:55:59,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-02-11 03:55:59,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-02-11 03:55:59,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-02-11 03:55:59,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-02-11 03:55:59,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-02-11 03:55:59,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-02-11 03:55:59,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-02-11 03:55:59,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-02-11 03:55:59,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-02-11 03:55:59,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-02-11 03:55:59,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-02-11 03:55:59,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-02-11 03:55:59,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-02-11 03:55:59,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-02-11 03:55:59,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-02-11 03:55:59,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-02-11 03:55:59,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-02-11 03:55:59,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-02-11 03:55:59,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-02-11 03:55:59,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-02-11 03:55:59,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-02-11 03:55:59,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-02-11 03:55:59,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-02-11 03:55:59,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-02-11 03:55:59,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-02-11 03:55:59,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-02-11 03:55:59,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-02-11 03:55:59,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-02-11 03:55:59,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-02-11 03:55:59,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-02-11 03:55:59,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-02-11 03:55:59,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-02-11 03:55:59,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-02-11 03:55:59,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-02-11 03:55:59,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-02-11 03:55:59,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-02-11 03:55:59,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-02-11 03:55:59,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-02-11 03:55:59,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-02-11 03:55:59,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-02-11 03:55:59,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-02-11 03:55:59,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-02-11 03:55:59,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-02-11 03:55:59,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-02-11 03:55:59,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-02-11 03:55:59,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-02-11 03:55:59,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-02-11 03:55:59,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-02-11 03:55:59,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-02-11 03:55:59,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-02-11 03:55:59,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-02-11 03:55:59,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-02-11 03:55:59,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-02-11 03:55:59,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-02-11 03:55:59,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-02-11 03:55:59,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-02-11 03:55:59,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-02-11 03:55:59,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-02-11 03:55:59,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-02-11 03:55:59,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-02-11 03:55:59,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-02-11 03:55:59,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-02-11 03:55:59,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-02-11 03:55:59,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-02-11 03:55:59,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-02-11 03:55:59,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-02-11 03:55:59,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-02-11 03:55:59,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-02-11 03:55:59,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-02-11 03:55:59,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-02-11 03:55:59,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-02-11 03:55:59,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-02-11 03:55:59,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-02-11 03:55:59,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-02-11 03:56:00,067 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-11 03:56:00,068 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-02-11 03:56:00,068 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-02-11 03:56:00,358 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 448 ms to try 5000 different iterations.  Found a solution that moves 4676 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1839877888195485. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038414778716574324, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05357897974679128, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5195555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038414778716574324, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17933333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-11 03:56:00,369 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1277996084:81 , srv700811291:81 , srv770284184:81 , srv848966791:81 , srv318205916:82 , srv601589319:82 , srv705085369:82 , srv810948958:82 , srv1429830901:83 , srv15395759:83 , srv1558186916:83 , srv1861314718:83 , srv20356281:83 , srv312383356:83 , srv997221324:83 , srv1205914559:84 , srv1698375738:84 , srv1752795630:84 , srv2054891350:84 , srv239924548:84 , srv299668954:84 , srv482838438:84 , srv515900047:84 , srv635201423:84 , srv985912693:84 , srv1005540877:85 , srv1461996381:85 , srv1485775565:85 , srv1730306016:85 , srv1974297545:85 , srv228592358:85 , srv341462400:85 , srv483602120:85 , srv894338769:85 , srv91265282:85 , srv92217382:85 , srv1191672231:86 , srv1216909676:86 , srv1273302204:86 , srv1412563751:86 , srv1479574219:86 , srv157937333:86 , srv1610487780:86 , srv1696939577:86 , srv1787702660:86 , srv1967885778:86 , srv2103969816:86 , srv277606235:86 , srv658908385:86 , srv681726610:86 , srv7026039:86 , srv812452801:86 , srv1581920051:87 , srv1654068194:87 , srv1748327787:87 , srv2001254925:87 , srv2048017920:87 , srv205362884:87 , srv2072048533:87 , srv277165825:87 , srv28970393:87 , srv335553573:87 , srv450407410:87 , srv553042203:87 , srv1025220556:88 , srv1078738101:88 , srv1613624140:88 , srv1625865534:88 , srv1798220426:88 , srv550616423:88 , srv641764602:88 , srv768651621:88 , srv874130901:88 , srv1200224599:89 , srv1230299602:89 , srv1875377077:89 , srv1904459476:89 , srv56323706:89 , srv576105633:89 , srv753435890:89 , srv787304599:89 , srv791767047:89 , srv1150378456:90 , srv1365046903:90 , srv1671308341:90 , srv1726730302:90 , srv1786947528:90 , srv1803609856:90 , srv1958375183:90 , srv278163448:90 , srv342706440:90 , srv824264789:90 , srv874220531:90 , srv1011008458:91 , srv1265461773:91 , srv981258659:92 , srv33231229:93 , srv356727189:93 , srv177848805:98 , srv250882279:431 }
2023-02-11 03:56:00,373 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-02-11 03:56:00,374 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1787702660=36, srv1011008458=1, srv1875377077=40, srv1461996381=16, srv1967885778=43, srv312383356=60, srv810948958=88, srv1798220426=37, srv1752795630=33, srv812452801=89, srv177848805=34, srv1696939577=28, srv1479574219=17, srv894338769=94, srv450407410=67, srv2103969816=51, srv1265461773=10, srv299668954=59, srv981258659=97, srv2054891350=49, srv483602120=69, srv1786947528=35, srv56323706=73, srv1005540877=0, srv2001254925=45, srv515900047=70, srv874130901=92, srv1412563751=14, srv848966791=91, srv91265282=95, srv1974297545=44, srv342706440=65, srv15395759=19, srv277606235=56, srv239924548=53, srv2048017920=47, srv335553573=63, srv157937333=21, srv1191672231=5, srv550616423=71, srv1205914559=7, srv1671308341=27, srv635201423=76, srv705085369=82, srv985912693=98, srv482838438=68, srv681726610=79, srv874220531=93, srv753435890=83, srv1216909676=8, srv1958375183=42, srv824264789=90, srv1625865534=25, srv700811291=80, srv553042203=72, srv770284184=85, srv28970393=58, srv641764602=77, srv1610487780=23, srv1273302204=11, srv997221324=99, srv601589319=75, srv1698375738=29, srv341462400=64, srv1200224599=6, srv1613624140=24, srv277165825=55, srv1277996084=12, srv1654068194=26, srv1726730302=30, srv318205916=61, srv33231229=62, srv1429830901=15, srv787304599=86, srv356727189=66, srv1803609856=38, srv7026039=81, srv658908385=78, srv1748327787=32, srv1581920051=22, srv1730306016=31, srv1150378456=4, srv228592358=52, srv1078738101=3, srv278163448=57, srv92217382=96, srv205362884=48, srv20356281=46, srv791767047=87, srv576105633=74, srv1861314718=39, srv1485775565=18, srv1025220556=2, srv1558186916=20, srv768651621=84, srv1904459476=41, srv2072048533=50, srv1365046903=13, srv250882279=54, srv1230299602=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-02-11 03:56:00,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-02-11 03:56:00,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-02-11 03:56:00,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-02-11 03:56:00,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-02-11 03:56:00,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-02-11 03:56:00,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-02-11 03:56:00,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-02-11 03:56:00,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-02-11 03:56:00,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-02-11 03:56:00,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-02-11 03:56:00,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-02-11 03:56:00,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-02-11 03:56:00,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-02-11 03:56:00,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-02-11 03:56:00,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-02-11 03:56:00,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-02-11 03:56:00,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-02-11 03:56:00,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-02-11 03:56:00,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-02-11 03:56:00,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-02-11 03:56:00,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-02-11 03:56:00,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-02-11 03:56:00,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-02-11 03:56:00,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-02-11 03:56:00,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-02-11 03:56:00,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-02-11 03:56:00,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-02-11 03:56:00,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-02-11 03:56:00,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-02-11 03:56:00,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-02-11 03:56:00,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-02-11 03:56:00,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-02-11 03:56:00,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-02-11 03:56:00,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-02-11 03:56:00,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-02-11 03:56:00,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-02-11 03:56:00,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-02-11 03:56:00,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-02-11 03:56:00,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-02-11 03:56:00,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-02-11 03:56:00,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-02-11 03:56:00,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-02-11 03:56:00,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-02-11 03:56:00,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-02-11 03:56:00,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-02-11 03:56:00,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-02-11 03:56:00,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-02-11 03:56:00,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-02-11 03:56:00,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-02-11 03:56:00,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-02-11 03:56:00,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-02-11 03:56:00,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-02-11 03:56:00,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-02-11 03:56:00,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-02-11 03:56:00,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-02-11 03:56:00,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-02-11 03:56:00,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-02-11 03:56:00,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-02-11 03:56:00,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-02-11 03:56:00,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-02-11 03:56:00,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-02-11 03:56:00,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-02-11 03:56:00,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-02-11 03:56:00,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-02-11 03:56:00,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-02-11 03:56:00,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-02-11 03:56:00,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-02-11 03:56:00,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-02-11 03:56:00,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-02-11 03:56:00,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-02-11 03:56:00,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-02-11 03:56:00,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-02-11 03:56:00,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-02-11 03:56:00,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-02-11 03:56:00,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-02-11 03:56:00,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-02-11 03:56:00,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-02-11 03:56:00,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-02-11 03:56:00,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-02-11 03:56:00,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-02-11 03:56:00,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-02-11 03:56:00,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-02-11 03:56:00,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-02-11 03:56:00,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-02-11 03:56:00,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-02-11 03:56:00,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-02-11 03:56:00,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-02-11 03:56:00,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-02-11 03:56:00,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-02-11 03:56:00,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-02-11 03:56:00,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-02-11 03:56:00,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-02-11 03:56:00,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-02-11 03:56:00,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-02-11 03:56:00,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-02-11 03:56:00,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-02-11 03:56:00,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-02-11 03:56:00,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-02-11 03:56:00,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-02-11 03:56:00,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-02-11 03:56:00,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-02-11 03:56:00,510 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038414778716574324, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05357897974679128, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038414778716574324, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17933333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-11 03:56:00,510 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-02-11 03:56:00,510 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18395504676773858, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038414778716574324, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05357897974679128, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.038414778716574324, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17933333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24525, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-02-11 03:56:00,768 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 359 ms to try 5000 different iterations.  Found a solution that moves 2743 regions; Going from a computed imbalance of 0.18395504676773858 to a new imbalance of 0.016103995104359444. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021959161270099663); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.028137070299305476, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.30477777777777776, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021959161270099663); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-11 03:56:00,777 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv177848805:74 , srv33231229:77 , srv1011008458:79 , srv1265461773:79 , srv356727189:80 , srv1958375183:81 , srv981258659:81 , srv1025220556:82 , srv1150378456:82 , srv1200224599:82 , srv1230299602:82 , srv1726730302:82 , srv342706440:82 , srv874220531:82 , srv1078738101:83 , srv1365046903:83 , srv1581920051:83 , srv1786947528:83 , srv1798220426:83 , srv1803609856:83 , srv278163448:83 , srv641764602:83 , srv787304599:83 , srv824264789:83 , srv1191672231:84 , srv1479574219:84 , srv1610487780:84 , srv1625865534:84 , srv1671308341:84 , srv1748327787:84 , srv1787702660:84 , srv1875377077:84 , srv1967885778:84 , srv205362884:84 , srv2072048533:84 , srv28970393:84 , srv450407410:84 , srv550616423:84 , srv56323706:84 , srv658908385:84 , srv753435890:84 , srv791767047:84 , srv1485775565:85 , srv157937333:85 , srv1654068194:85 , srv1904459476:85 , srv1974297545:85 , srv277165825:85 , srv768651621:85 , srv812452801:85 , srv92217382:85 , srv1216909676:86 , srv1412563751:86 , srv1696939577:86 , srv2001254925:86 , srv2048017920:86 , srv299668954:86 , srv335553573:86 , srv515900047:86 , srv894338769:86 , srv1429830901:87 , srv1461996381:87 , srv1613624140:87 , srv1698375738:87 , srv1730306016:87 , srv2103969816:87 , srv239924548:87 , srv277606235:87 , srv341462400:87 , srv576105633:87 , srv681726610:87 , srv7026039:87 , srv985912693:87 , srv1205914559:88 , srv15395759:88 , srv1752795630:88 , srv1861314718:88 , srv20356281:88 , srv2054891350:88 , srv228592358:88 , srv312383356:88 , srv553042203:88 , srv1273302204:89 , srv1558186916:89 , srv318205916:89 , srv483602120:89 , srv1005540877:90 , srv482838438:90 , srv601589319:90 , srv635201423:90 , srv700811291:90 , srv705085369:90 , srv810948958:90 , srv848966791:90 , srv91265282:90 , srv1277996084:91 , srv874130901:91 , srv770284184:92 , srv997221324:93 , srv250882279:538 }
2023-02-11 03:56:00,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-02-11 03:56:00,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1787702660=36, srv1011008458=1, srv1875377077=40, srv1461996381=16, srv1967885778=43, srv312383356=60, srv810948958=88, srv1798220426=37, srv1752795630=33, srv812452801=89, srv177848805=34, srv1696939577=28, srv1479574219=17, srv894338769=94, srv450407410=67, srv2103969816=51, srv1265461773=10, srv299668954=59, srv981258659=97, srv2054891350=49, srv483602120=69, srv1786947528=35, srv56323706=73, srv1005540877=0, srv2001254925=45, srv515900047=70, srv874130901=92, srv1412563751=14, srv848966791=91, srv91265282=95, srv1974297545=44, srv342706440=65, srv15395759=19, srv277606235=56, srv239924548=53, srv2048017920=47, srv335553573=63, srv157937333=21, srv1191672231=5, srv550616423=71, srv1205914559=7, srv1671308341=27, srv635201423=76, srv705085369=82, srv985912693=98, srv482838438=68, srv681726610=79, srv874220531=93, srv753435890=83, srv1216909676=8, srv1958375183=42, srv824264789=90, srv1625865534=25, srv700811291=80, srv553042203=72, srv770284184=85, srv28970393=58, srv641764602=77, srv1610487780=23, srv1273302204=11, srv997221324=99, srv601589319=75, srv1698375738=29, srv341462400=64, srv1200224599=6, srv1613624140=24, srv277165825=55, srv1277996084=12, srv1654068194=26, srv1726730302=30, srv318205916=61, srv33231229=62, srv1429830901=15, srv787304599=86, srv356727189=66, srv1803609856=38, srv7026039=81, srv658908385=78, srv1748327787=32, srv1581920051=22, srv1730306016=31, srv1150378456=4, srv228592358=52, srv1078738101=3, srv278163448=57, srv92217382=96, srv205362884=48, srv20356281=46, srv791767047=87, srv576105633=74, srv1861314718=39, srv1485775565=18, srv1025220556=2, srv1558186916=20, srv768651621=84, srv1904459476=41, srv2072048533=50, srv1365046903=13, srv250882279=54, srv1230299602=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-02-11 03:56:00,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-02-11 03:56:00,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-02-11 03:56:00,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-02-11 03:56:00,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-02-11 03:56:00,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-02-11 03:56:00,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-02-11 03:56:00,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-02-11 03:56:00,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-02-11 03:56:00,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-02-11 03:56:00,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-02-11 03:56:00,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-02-11 03:56:00,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-02-11 03:56:00,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-02-11 03:56:00,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-02-11 03:56:00,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-02-11 03:56:00,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-02-11 03:56:00,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-02-11 03:56:00,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-02-11 03:56:00,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-02-11 03:56:00,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-02-11 03:56:00,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-02-11 03:56:00,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-02-11 03:56:00,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-02-11 03:56:00,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-02-11 03:56:00,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-02-11 03:56:00,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-02-11 03:56:00,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-02-11 03:56:00,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-02-11 03:56:00,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-02-11 03:56:00,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-02-11 03:56:00,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-02-11 03:56:00,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-02-11 03:56:00,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-02-11 03:56:00,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-02-11 03:56:00,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-02-11 03:56:00,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-02-11 03:56:00,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-02-11 03:56:00,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-02-11 03:56:00,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-02-11 03:56:00,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-02-11 03:56:00,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-02-11 03:56:00,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-02-11 03:56:00,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-02-11 03:56:00,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-02-11 03:56:00,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-02-11 03:56:00,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-02-11 03:56:00,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-02-11 03:56:00,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-02-11 03:56:00,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-02-11 03:56:00,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-02-11 03:56:00,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-02-11 03:56:00,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-02-11 03:56:00,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-02-11 03:56:00,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-02-11 03:56:00,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-02-11 03:56:00,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-02-11 03:56:00,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-02-11 03:56:00,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-02-11 03:56:00,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-02-11 03:56:00,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-02-11 03:56:00,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-02-11 03:56:00,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-02-11 03:56:00,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-02-11 03:56:00,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-02-11 03:56:00,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-02-11 03:56:00,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-02-11 03:56:00,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-02-11 03:56:00,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-02-11 03:56:00,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-02-11 03:56:00,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-02-11 03:56:00,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-02-11 03:56:00,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-02-11 03:56:00,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-02-11 03:56:00,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-02-11 03:56:00,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-02-11 03:56:00,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-02-11 03:56:00,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-02-11 03:56:00,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-02-11 03:56:00,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-02-11 03:56:00,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-02-11 03:56:00,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-02-11 03:56:00,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-02-11 03:56:00,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-02-11 03:56:00,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-02-11 03:56:00,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-02-11 03:56:00,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-02-11 03:56:00,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-02-11 03:56:00,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-02-11 03:56:00,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-02-11 03:56:00,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-02-11 03:56:00,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-02-11 03:56:00,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-02-11 03:56:00,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-02-11 03:56:00,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-02-11 03:56:00,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-02-11 03:56:00,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-02-11 03:56:00,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-02-11 03:56:00,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-02-11 03:56:00,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-02-11 03:56:00,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-02-11 03:56:00,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-02-11 03:56:00,917 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021959161270099663); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.028137070299305476, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021959161270099663); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-11 03:56:00,917 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-02-11 03:56:00,917 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016084788207842215, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021959161270099663); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.028137070299305476, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021959161270099663); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013166666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04441666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-02-11 03:56:01,171 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 349 ms to try 5000 different iterations.  Found a solution that moves 984 regions; Going from a computed imbalance of 0.016084788207842215 to a new imbalance of 2.4327554784938905E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006436350813697321); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007293740686277048); MoveCostFunction : (multiplier=7.0, imbalance=0.10933333333333334, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006436350813697321); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0019166666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-11 03:56:01,174 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv874130901:80 , srv1005540877:82 , srv997221324:82 , srv1273302204:83 , srv1671308341:83 , srv483602120:83 , srv576105633:83 , srv7026039:83 , srv91265282:83 , srv1613624140:84 , srv1786947528:84 , srv1875377077:84 , srv1904459476:84 , srv2103969816:84 , srv335553573:84 , srv356727189:84 , srv482838438:84 , srv553042203:84 , srv635201423:84 , srv681726610:84 , srv705085369:84 , srv753435890:84 , srv824264789:84 , srv981258659:84 , srv1205914559:85 , srv1277996084:85 , srv1365046903:85 , srv1412563751:85 , srv1558186916:85 , srv1696939577:85 , srv1726730302:85 , srv1730306016:85 , srv1752795630:85 , srv177848805:85 , srv1803609856:85 , srv2001254925:85 , srv2048017920:85 , srv2054891350:85 , srv228592358:85 , srv277606235:85 , srv278163448:85 , srv341462400:85 , srv342706440:85 , srv56323706:85 , srv601589319:85 , srv641764602:85 , srv768651621:85 , srv770284184:85 , srv787304599:85 , srv791767047:85 , srv810948958:85 , srv874220531:85 , srv1078738101:86 , srv1150378456:86 , srv1200224599:86 , srv1216909676:86 , srv1230299602:86 , srv1461996381:86 , srv15395759:86 , srv1654068194:86 , srv1698375738:86 , srv1748327787:86 , srv1958375183:86 , srv20356281:86 , srv205362884:86 , srv239924548:86 , srv277165825:86 , srv28970393:86 , srv450407410:86 , srv550616423:86 , srv700811291:86 , srv848966791:86 , srv894338769:86 , srv985912693:86 , srv1011008458:87 , srv1025220556:87 , srv1429830901:87 , srv1479574219:87 , srv1485775565:87 , srv157937333:87 , srv1581920051:87 , srv1610487780:87 , srv1625865534:87 , srv1798220426:87 , srv1861314718:87 , srv1967885778:87 , srv1974297545:87 , srv2072048533:87 , srv312383356:87 , srv318205916:87 , srv33231229:87 , srv515900047:87 , srv658908385:87 , srv812452801:87 , srv1191672231:88 , srv1265461773:88 , srv1787702660:88 , srv299668954:88 , srv92217382:88 , srv250882279:546 }
2023-02-11 03:56:01,177 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-02-11 03:56:01,178 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1787702660=36, srv1011008458=1, srv1875377077=40, srv1461996381=16, srv1967885778=43, srv312383356=60, srv810948958=88, srv1798220426=37, srv1752795630=33, srv812452801=89, srv177848805=34, srv1696939577=28, srv1479574219=17, srv894338769=94, srv450407410=67, srv2103969816=51, srv1265461773=10, srv299668954=59, srv981258659=97, srv2054891350=49, srv483602120=69, srv1786947528=35, srv56323706=73, srv1005540877=0, srv2001254925=45, srv515900047=70, srv874130901=92, srv1412563751=14, srv848966791=91, srv91265282=95, srv1974297545=44, srv342706440=65, srv15395759=19, srv277606235=56, srv239924548=53, srv2048017920=47, srv335553573=63, srv157937333=21, srv1191672231=5, srv550616423=71, srv1205914559=7, srv1671308341=27, srv635201423=76, srv705085369=82, srv985912693=98, srv482838438=68, srv681726610=79, srv874220531=93, srv753435890=83, srv1216909676=8, srv1958375183=42, srv824264789=90, srv1625865534=25, srv700811291=80, srv553042203=72, srv770284184=85, srv28970393=58, srv641764602=77, srv1610487780=23, srv1273302204=11, srv997221324=99, srv601589319=75, srv1698375738=29, srv341462400=64, srv1200224599=6, srv1613624140=24, srv277165825=55, srv1277996084=12, srv1654068194=26, srv1726730302=30, srv318205916=61, srv33231229=62, srv1429830901=15, srv787304599=86, srv356727189=66, srv1803609856=38, srv7026039=81, srv658908385=78, srv1748327787=32, srv1581920051=22, srv1730306016=31, srv1150378456=4, srv228592358=52, srv1078738101=3, srv278163448=57, srv92217382=96, srv205362884=48, srv20356281=46, srv791767047=87, srv576105633=74, srv1861314718=39, srv1485775565=18, srv1025220556=2, srv1558186916=20, srv768651621=84, srv1904459476=41, srv2072048533=50, srv1365046903=13, srv250882279=54, srv1230299602=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-02-11 03:56:01,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-02-11 03:56:01,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-02-11 03:56:01,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-02-11 03:56:01,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-02-11 03:56:01,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-02-11 03:56:01,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-02-11 03:56:01,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-02-11 03:56:01,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-02-11 03:56:01,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-02-11 03:56:01,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-02-11 03:56:01,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-02-11 03:56:01,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-02-11 03:56:01,190 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-02-11 03:56:01,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-02-11 03:56:01,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-02-11 03:56:01,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-02-11 03:56:01,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-02-11 03:56:01,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-02-11 03:56:01,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-02-11 03:56:01,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-02-11 03:56:01,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-02-11 03:56:01,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-02-11 03:56:01,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-02-11 03:56:01,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-02-11 03:56:01,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-02-11 03:56:01,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-02-11 03:56:01,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-02-11 03:56:01,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-02-11 03:56:01,191 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-02-11 03:56:01,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-02-11 03:56:01,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-02-11 03:56:01,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-02-11 03:56:01,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-02-11 03:56:01,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-02-11 03:56:01,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-02-11 03:56:01,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-02-11 03:56:01,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-02-11 03:56:01,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-02-11 03:56:01,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-02-11 03:56:01,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-02-11 03:56:01,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-02-11 03:56:01,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-02-11 03:56:01,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-02-11 03:56:01,192 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-02-11 03:56:01,193 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-02-11 03:56:01,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-02-11 03:56:01,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-02-11 03:56:01,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-02-11 03:56:01,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-02-11 03:56:01,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-02-11 03:56:01,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-02-11 03:56:01,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-02-11 03:56:01,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-02-11 03:56:01,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-02-11 03:56:01,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-02-11 03:56:01,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-02-11 03:56:01,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-02-11 03:56:01,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-02-11 03:56:01,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-02-11 03:56:01,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-02-11 03:56:01,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-02-11 03:56:01,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-02-11 03:56:01,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-02-11 03:56:01,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-02-11 03:56:01,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-02-11 03:56:01,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-02-11 03:56:01,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-02-11 03:56:01,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-02-11 03:56:01,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-02-11 03:56:01,300 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006436350813697321); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007293740686277048); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006436350813697321); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0019166666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-11 03:56:01,300 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-02-11 03:56:01,301 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.363854325840026E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006436350813697321); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007293740686277048); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006436350813697321); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0019166666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-02-11 03:56:01,638 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 430 ms to try 5000 different iterations.  Found a solution that moves 220 regions; Going from a computed imbalance of 2.363854325840026E-4 to a new imbalance of 1.4732115219397209E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.735366466207394E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0026378918903435544); MoveCostFunction : (multiplier=7.0, imbalance=0.024444444444444446); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.735366466207394E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-11 03:56:01,640 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1613624140:83 , srv1625865534:83 , srv2072048533:83 , srv553042203:83 , srv576105633:83 , srv874130901:83 , srv1150378456:84 , srv1191672231:84 , srv1200224599:84 , srv1265461773:84 , srv1273302204:84 , srv1365046903:84 , srv1461996381:84 , srv157937333:84 , srv1654068194:84 , srv1726730302:84 , srv1787702660:84 , srv1798220426:84 , srv1803609856:84 , srv1861314718:84 , srv1904459476:84 , srv2001254925:84 , srv2048017920:84 , srv228592358:84 , srv277165825:84 , srv277606235:84 , srv278163448:84 , srv299668954:84 , srv312383356:84 , srv318205916:84 , srv482838438:84 , srv56323706:84 , srv635201423:84 , srv768651621:84 , srv770284184:84 , srv791767047:84 , srv812452801:84 , srv91265282:84 , srv92217382:84 , srv997221324:84 , srv1005540877:85 , srv1011008458:85 , srv1025220556:85 , srv1078738101:85 , srv1205914559:85 , srv1216909676:85 , srv1230299602:85 , srv1277996084:85 , srv1412563751:85 , srv1429830901:85 , srv1479574219:85 , srv1485775565:85 , srv15395759:85 , srv1558186916:85 , srv1581920051:85 , srv1610487780:85 , srv1671308341:85 , srv1696939577:85 , srv1698375738:85 , srv1730306016:85 , srv1748327787:85 , srv1752795630:85 , srv177848805:85 , srv1786947528:85 , srv1875377077:85 , srv1958375183:85 , srv1967885778:85 , srv1974297545:85 , srv20356281:85 , srv205362884:85 , srv2054891350:85 , srv2103969816:85 , srv239924548:85 , srv28970393:85 , srv33231229:85 , srv335553573:85 , srv341462400:85 , srv342706440:85 , srv356727189:85 , srv450407410:85 , srv483602120:85 , srv515900047:85 , srv550616423:85 , srv601589319:85 , srv658908385:85 , srv681726610:85 , srv700811291:85 , srv787304599:85 , srv810948958:85 , srv824264789:85 , srv848966791:85 , srv874220531:85 , srv894338769:85 , srv981258659:85 , srv985912693:85 , srv641764602:86 , srv7026039:86 , srv705085369:86 , srv753435890:86 , srv250882279:627 }
2023-02-11 03:56:01,645 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-02-11 03:56:01,646 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1787702660=36, srv1011008458=1, srv1875377077=40, srv1461996381=16, srv1967885778=43, srv312383356=60, srv810948958=88, srv1798220426=37, srv1752795630=33, srv812452801=89, srv177848805=34, srv1696939577=28, srv1479574219=17, srv894338769=94, srv450407410=67, srv2103969816=51, srv1265461773=10, srv299668954=59, srv981258659=97, srv2054891350=49, srv483602120=69, srv1786947528=35, srv56323706=73, srv1005540877=0, srv2001254925=45, srv515900047=70, srv874130901=92, srv1412563751=14, srv848966791=91, srv91265282=95, srv1974297545=44, srv342706440=65, srv15395759=19, srv277606235=56, srv239924548=53, srv2048017920=47, srv335553573=63, srv157937333=21, srv1191672231=5, srv550616423=71, srv1205914559=7, srv1671308341=27, srv635201423=76, srv705085369=82, srv985912693=98, srv482838438=68, srv681726610=79, srv874220531=93, srv753435890=83, srv1216909676=8, srv1958375183=42, srv824264789=90, srv1625865534=25, srv700811291=80, srv553042203=72, srv770284184=85, srv28970393=58, srv641764602=77, srv1610487780=23, srv1273302204=11, srv997221324=99, srv601589319=75, srv1698375738=29, srv341462400=64, srv1200224599=6, srv1613624140=24, srv277165825=55, srv1277996084=12, srv1654068194=26, srv1726730302=30, srv318205916=61, srv33231229=62, srv1429830901=15, srv787304599=86, srv356727189=66, srv1803609856=38, srv7026039=81, srv658908385=78, srv1748327787=32, srv1581920051=22, srv1730306016=31, srv1150378456=4, srv228592358=52, srv1078738101=3, srv278163448=57, srv92217382=96, srv205362884=48, srv20356281=46, srv791767047=87, srv576105633=74, srv1861314718=39, srv1485775565=18, srv1025220556=2, srv1558186916=20, srv768651621=84, srv1904459476=41, srv2072048533=50, srv1365046903=13, srv250882279=54, srv1230299602=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-02-11 03:56:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-02-11 03:56:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-02-11 03:56:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-02-11 03:56:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-02-11 03:56:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-02-11 03:56:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-02-11 03:56:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-02-11 03:56:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-02-11 03:56:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-02-11 03:56:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-02-11 03:56:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-02-11 03:56:01,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-02-11 03:56:01,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-02-11 03:56:01,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-02-11 03:56:01,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-02-11 03:56:01,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-02-11 03:56:01,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-02-11 03:56:01,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-02-11 03:56:01,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-02-11 03:56:01,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-02-11 03:56:01,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-02-11 03:56:01,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-02-11 03:56:01,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-02-11 03:56:01,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-02-11 03:56:01,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-02-11 03:56:01,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-02-11 03:56:01,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-02-11 03:56:01,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-02-11 03:56:01,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-02-11 03:56:01,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-02-11 03:56:01,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-02-11 03:56:01,771 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.3191642294209175E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=2.735366466207394E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0026378918903435544); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=2.735366466207394E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-11 03:56:01,771 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1613624140:83 , srv1625865534:83 , srv2072048533:83 , srv553042203:83 , srv576105633:83 , srv874130901:83 , srv1150378456:84 , srv1191672231:84 , srv1200224599:84 , srv1265461773:84 , srv1273302204:84 , srv1365046903:84 , srv1461996381:84 , srv157937333:84 , srv1654068194:84 , srv1726730302:84 , srv1787702660:84 , srv1798220426:84 , srv1803609856:84 , srv1861314718:84 , srv1904459476:84 , srv2001254925:84 , srv2048017920:84 , srv228592358:84 , srv277165825:84 , srv277606235:84 , srv278163448:84 , srv299668954:84 , srv312383356:84 , srv318205916:84 , srv482838438:84 , srv56323706:84 , srv635201423:84 , srv768651621:84 , srv770284184:84 , srv791767047:84 , srv812452801:84 , srv91265282:84 , srv92217382:84 , srv997221324:84 , srv1005540877:85 , srv1011008458:85 , srv1025220556:85 , srv1078738101:85 , srv1205914559:85 , srv1216909676:85 , srv1230299602:85 , srv1277996084:85 , srv1412563751:85 , srv1429830901:85 , srv1479574219:85 , srv1485775565:85 , srv15395759:85 , srv1558186916:85 , srv1581920051:85 , srv1610487780:85 , srv1671308341:85 , srv1696939577:85 , srv1698375738:85 , srv1730306016:85 , srv1748327787:85 , srv1752795630:85 , srv177848805:85 , srv1786947528:85 , srv1875377077:85 , srv1958375183:85 , srv1967885778:85 , srv1974297545:85 , srv20356281:85 , srv205362884:85 , srv2054891350:85 , srv2103969816:85 , srv239924548:85 , srv28970393:85 , srv33231229:85 , srv335553573:85 , srv341462400:85 , srv342706440:85 , srv356727189:85 , srv450407410:85 , srv483602120:85 , srv515900047:85 , srv550616423:85 , srv601589319:85 , srv658908385:85 , srv681726610:85 , srv700811291:85 , srv787304599:85 , srv810948958:85 , srv824264789:85 , srv848966791:85 , srv874220531:85 , srv894338769:85 , srv981258659:85 , srv985912693:85 , srv641764602:86 , srv7026039:86 , srv705085369:86 , srv753435890:86 , srv250882279:627 }
2023-02-11 03:56:01,798 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=132 (was 109) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=3283 (was 3765)
2023-02-11 03:56:01,806 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=132, ProcessCount=168, AvailableMemoryMB=3282
2023-02-11 03:56:01,807 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-02-11 03:56:01,807 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-02-11 03:56:01,815 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1065704518:84 , srv1104295365:84 , srv1185834713:84 , srv1415728526:84 , srv143113487:84 , srv1444959856:84 , srv1478865302:84 , srv1551309457:84 , srv1579879036:84 , srv1764702279:84 , srv1826717289:84 , srv2012306029:84 , srv2018084329:84 , srv2074344118:84 , srv218169980:84 , srv245927047:84 , srv311116006:84 , srv394403636:84 , srv426902364:84 , srv483074906:84 , srv53440228:84 , srv541918130:84 , srv576630662:84 , srv622161116:84 , srv626506199:84 , srv629309043:84 , srv67822220:84 , srv75199624:84 , srv908001229:84 , srv747546633:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-02-11 03:56:01,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-02-11 03:56:01,816 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1551309457=7, srv541918130=21, srv143113487=4, srv1826717289=10, srv1579879036=8, srv426902364=18, srv622161116=23, srv1415728526=3, srv1065704518=0, srv2012306029=11, srv1444959856=5, srv1185834713=2, srv576630662=22, srv483074906=19, srv908001229=29, srv1764702279=9, srv311116006=16, srv67822220=26, srv1104295365=1, srv629309043=25, srv1478865302=6, srv747546633=27, srv2018084329=12, srv626506199=24, srv245927047=15, srv2074344118=13, srv394403636=17, srv53440228=20, srv218169980=14, srv75199624=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-02-11 03:56:01,823 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-02-11 03:56:01,824 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-02-11 03:56:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-02-11 03:56:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-02-11 03:56:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-02-11 03:56:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-02-11 03:56:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-02-11 03:56:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-02-11 03:56:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-02-11 03:56:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-02-11 03:56:01,825 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-02-11 03:56:01,840 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1311911718401844, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-11 03:56:01,840 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912606422362962, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1311911718401844, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-02-11 03:58:01,828 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6259959 different iterations.  Found a solution that moves 1848 regions; Going from a computed imbalance of 0.9912606422362962 to a new imbalance of 4.313324190526491E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6844444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-11 03:58:01,836 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1065704518:90 , srv1104295365:90 , srv1185834713:90 , srv1415728526:90 , srv143113487:90 , srv1444959856:90 , srv1478865302:90 , srv1551309457:90 , srv1579879036:90 , srv1764702279:90 , srv1826717289:90 , srv2012306029:90 , srv2018084329:90 , srv2074344118:90 , srv218169980:90 , srv245927047:90 , srv311116006:90 , srv394403636:90 , srv426902364:90 , srv483074906:90 , srv53440228:90 , srv541918130:90 , srv576630662:90 , srv622161116:90 , srv626506199:90 , srv629309043:90 , srv67822220:90 , srv747546633:90 , srv75199624:90 , srv908001229:90 }
2023-02-11 03:58:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-02-11 03:58:01,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1551309457=7, srv541918130=21, srv143113487=4, srv1826717289=10, srv1579879036=8, srv426902364=18, srv622161116=23, srv1415728526=3, srv1065704518=0, srv2012306029=11, srv1444959856=5, srv1185834713=2, srv576630662=22, srv483074906=19, srv908001229=29, srv1764702279=9, srv311116006=16, srv67822220=26, srv1104295365=1, srv629309043=25, srv1478865302=6, srv747546633=27, srv2018084329=12, srv626506199=24, srv245927047=15, srv2074344118=13, srv394403636=17, srv53440228=20, srv218169980=14, srv75199624=28} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-02-11 03:58:01,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-02-11 03:58:01,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-02-11 03:58:01,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-02-11 03:58:01,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-02-11 03:58:01,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-02-11 03:58:01,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-02-11 03:58:01,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-02-11 03:58:01,842 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-02-11 03:58:01,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-02-11 03:58:01,843 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-02-11 03:58:01,856 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-02-11 03:58:01,856 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1065704518:90 , srv1104295365:90 , srv1185834713:90 , srv1415728526:90 , srv143113487:90 , srv1444959856:90 , srv1478865302:90 , srv1551309457:90 , srv1579879036:90 , srv1764702279:90 , srv1826717289:90 , srv2012306029:90 , srv2018084329:90 , srv2074344118:90 , srv218169980:90 , srv245927047:90 , srv311116006:90 , srv394403636:90 , srv426902364:90 , srv483074906:90 , srv53440228:90 , srv541918130:90 , srv576630662:90 , srv622161116:90 , srv626506199:90 , srv629309043:90 , srv67822220:90 , srv747546633:90 , srv75199624:90 , srv908001229:90 }
2023-02-11 03:58:01,869 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=323 (was 132) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 168) - ProcessCount LEAK? -, AvailableMemoryMB=3046 (was 3282)