2023-11-20 08:55:24,204 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-11-20 08:55:24,478 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-11-20 08:55:24,590 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-11-20 08:55:24,653 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-20 08:55:24,660 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-20 08:55:24,673 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=93, ProcessCount=168, AvailableMemoryMB=15648
2023-11-20 08:55:24,676 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-20 08:55:24,676 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-11-20 08:55:24,799 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1030653345:84 , srv1054233374:84 , srv1058249223:84 , srv1070885059:84 , srv1085821952:84 , srv112741249:84 , srv1138765525:84 , srv1162731849:84 , srv120480784:84 , srv1216846790:84 , srv1222917565:84 , srv1233047650:84 , srv1314045301:84 , srv1320867707:84 , srv1330143588:84 , srv1371794176:84 , srv1435407348:84 , srv1456399456:84 , srv1456948942:84 , srv1486634619:84 , srv1512079295:84 , srv152571948:84 , srv1532633762:84 , srv155623916:84 , srv1556331307:84 , srv1601920551:84 , srv1631268622:84 , srv1662903068:84 , srv1664319662:84 , srv1672408131:84 , srv1674170258:84 , srv1705575887:84 , srv1755925672:84 , srv1758296038:84 , srv1817124160:84 , srv1822406739:84 , srv1845606471:84 , srv1855188233:84 , srv1858106515:84 , srv1889134791:84 , srv1905030220:84 , srv1926640988:84 , srv1929152920:84 , srv2004596344:84 , srv2013109571:84 , srv2018013485:84 , srv2022037757:84 , srv2048436983:84 , srv2053155197:84 , srv2053264706:84 , srv2068231608:84 , srv2091754783:84 , srv2106832016:84 , srv213867589:84 , srv2141021208:84 , srv267720778:84 , srv276821199:84 , srv281652241:84 , srv286794869:84 , srv309625420:84 , srv313012972:84 , srv313660382:84 , srv357808633:84 , srv358796505:84 , srv398588155:84 , srv440192557:84 , srv443351923:84 , srv447695486:84 , srv457442417:84 , srv460209325:84 , srv46134110:84 , srv470979460:84 , srv501555138:84 , srv59194775:84 , srv600317870:84 , srv646543227:84 , srv646784755:84 , srv647891130:84 , srv675431559:84 , srv760416909:84 , srv779373093:84 , srv785404383:84 , srv795690327:84 , srv804065926:84 , srv805462027:84 , srv805578977:84 , srv818163004:84 , srv820016866:84 , srv825448834:84 , srv851223456:84 , srv873378001:84 , srv880917271:84 , srv904753482:84 , srv91269851:84 , srv916124280:84 , srv916971801:84 , srv942792350:84 , srv971920447:84 , srv987495560:84 , srv2123211837:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-11-20 08:55:24,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-20 08:55:24,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1855188233=37, srv785404383=82, srv760416909=80, srv281652241=58, srv286794869=59, srv1486634619=19, srv1631268622=26, srv1233047650=11, srv1162731849=7, srv267720778=56, srv942792350=97, srv1222917565=10, srv2004596344=43, srv1320867707=13, srv1371794176=15, srv987495560=99, srv443351923=67, srv820016866=88, srv398588155=65, srv1672408131=29, srv120480784=8, srv1858106515=38, srv470979460=72, srv1845606471=36, srv2018013485=45, srv805578977=86, srv2123211837=53, srv112741249=5, srv2106832016=52, srv2013109571=44, srv1435407348=16, srv818163004=87, srv1054233374=1, srv460209325=70, srv1456399456=17, srv358796505=64, srv1755925672=32, srv795690327=83, srv155623916=23, srv1070885059=3, srv1664319662=28, srv1030653345=0, srv501555138=73, srv313012972=61, srv213867589=54, srv1674170258=30, srv1926640988=41, srv447695486=68, srv804065926=84, srv2141021208=55, srv873378001=91, srv646543227=76, srv1758296038=33, srv2091754783=51, srv904753482=93, srv647891130=78, srv457442417=69, srv1512079295=20, srv357808633=63, srv1138765525=6, srv1314045301=12, srv1058249223=2, srv646784755=77, srv309625420=60, srv1216846790=9, srv1532633762=22, srv1705575887=31, srv1905030220=40, srv46134110=71, srv2048436983=47, srv276821199=57, srv440192557=66, srv916124280=95, srv1929152920=42, srv916971801=96, srv2068231608=50, srv1085821952=4, srv825448834=89, srv805462027=85, srv2053264706=49, srv1330143588=14, srv1601920551=25, srv59194775=74, srv600317870=75, srv880917271=92, srv1456948942=18, srv1822406739=35, srv971920447=98, srv91269851=94, srv1662903068=27, srv2053155197=48, srv675431559=79, srv779373093=81, srv2022037757=46, srv1889134791=39, srv851223456=90, srv1556331307=24, srv1817124160=34, srv313660382=62, srv152571948=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-20 08:55:24,878 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-20 08:55:24,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-20 08:55:24,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-20 08:55:24,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-20 08:55:24,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-20 08:55:24,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-20 08:55:24,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-20 08:55:24,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-20 08:55:24,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-20 08:55:24,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-20 08:55:24,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-20 08:55:24,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-20 08:55:24,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-20 08:55:24,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-20 08:55:24,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-20 08:55:24,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-20 08:55:24,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-20 08:55:24,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-20 08:55:24,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-20 08:55:24,880 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-20 08:55:24,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-20 08:55:24,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-20 08:55:24,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-20 08:55:24,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-20 08:55:24,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-20 08:55:24,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-20 08:55:24,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-20 08:55:24,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-20 08:55:24,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-20 08:55:24,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-20 08:55:24,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-20 08:55:24,881 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-20 08:55:24,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-20 08:55:24,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-20 08:55:24,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-20 08:55:24,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-20 08:55:24,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-20 08:55:24,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-20 08:55:24,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-20 08:55:24,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-20 08:55:24,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-20 08:55:24,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-20 08:55:24,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-20 08:55:24,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-20 08:55:24,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-20 08:55:24,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-20 08:55:24,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-20 08:55:24,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-20 08:55:24,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-20 08:55:24,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-20 08:55:24,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-20 08:55:24,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-20 08:55:24,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-20 08:55:24,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-20 08:55:24,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-20 08:55:24,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-20 08:55:24,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-20 08:55:24,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-20 08:55:24,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-20 08:55:24,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-20 08:55:24,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-20 08:55:24,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-20 08:55:24,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-20 08:55:24,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-20 08:55:24,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-20 08:55:24,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-20 08:55:24,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-20 08:55:24,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-20 08:55:24,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-20 08:55:24,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-20 08:55:24,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-20 08:55:24,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-20 08:55:24,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-20 08:55:24,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-20 08:55:24,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-20 08:55:24,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-20 08:55:24,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-20 08:55:24,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-20 08:55:24,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-20 08:55:24,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-20 08:55:24,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-20 08:55:24,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-20 08:55:24,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-20 08:55:24,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-20 08:55:24,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-20 08:55:24,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-20 08:55:24,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-20 08:55:24,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-20 08:55:24,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-20 08:55:24,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-20 08:55:24,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-20 08:55:24,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-20 08:55:24,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-20 08:55:24,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-20 08:55:24,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-20 08:55:24,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-20 08:55:24,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-20 08:55:24,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-20 08:55:24,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-20 08:55:24,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-20 08:55:24,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-20 08:55:25,065 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-20 08:55:25,065 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-20 08:55:25,065 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-20 08:55:25,479 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 568 ms to try 5000 different iterations.  Found a solution that moves 4709 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17867703302253946. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03859937124205303, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0535244890824527, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5232222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03859937124205303, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-20 08:55:25,488 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1905030220:82 , srv460209325:82 , srv675431559:82 , srv785404383:82 , srv120480784:83 , srv1330143588:83 , srv1456948942:83 , srv1664319662:83 , srv2004596344:83 , srv276821199:83 , srv470979460:83 , srv59194775:83 , srv820016866:83 , srv916971801:83 , srv1486634619:84 , srv2022037757:84 , srv2106832016:84 , srv267720778:84 , srv309625420:84 , srv313660382:84 , srv646543227:84 , srv112741249:85 , srv1512079295:85 , srv155623916:85 , srv1556331307:85 , srv1662903068:85 , srv1755925672:85 , srv1858106515:85 , srv1929152920:85 , srv2048436983:85 , srv2053155197:85 , srv2091754783:85 , srv358796505:85 , srv818163004:85 , srv1162731849:86 , srv1371794176:86 , srv1674170258:86 , srv1705575887:86 , srv1817124160:86 , srv1822406739:86 , srv1845606471:86 , srv2013109571:86 , srv286794869:86 , srv357808633:86 , srv443351923:86 , srv646784755:86 , srv779373093:86 , srv804065926:86 , srv805462027:86 , srv851223456:86 , srv1030653345:87 , srv1222917565:87 , srv1758296038:87 , srv1855188233:87 , srv1889134791:87 , srv2018013485:87 , srv213867589:87 , srv281652241:87 , srv440192557:87 , srv447695486:87 , srv457442417:87 , srv600317870:87 , srv647891130:87 , srv91269851:87 , srv942792350:87 , srv987495560:87 , srv1085821952:88 , srv1216846790:88 , srv1314045301:88 , srv1435407348:88 , srv1532633762:88 , srv1601920551:88 , srv2068231608:88 , srv313012972:88 , srv760416909:88 , srv795690327:88 , srv805578977:88 , srv873378001:88 , srv904753482:88 , srv916124280:88 , srv1054233374:89 , srv1058249223:89 , srv1138765525:89 , srv1233047650:89 , srv1631268622:89 , srv398588155:89 , srv46134110:89 , srv825448834:89 , srv1320867707:90 , srv1926640988:90 , srv2053264706:90 , srv2141021208:90 , srv1456399456:91 , srv152571948:91 , srv501555138:91 , srv880917271:91 , srv1672408131:92 , srv1070885059:94 , srv971920447:94 , srv2123211837:433 }
2023-11-20 08:55:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-20 08:55:25,492 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1855188233=37, srv785404383=82, srv760416909=80, srv281652241=58, srv286794869=59, srv1486634619=19, srv1631268622=26, srv1233047650=11, srv1162731849=7, srv267720778=56, srv942792350=97, srv1222917565=10, srv2004596344=43, srv1320867707=13, srv1371794176=15, srv987495560=99, srv443351923=67, srv820016866=88, srv398588155=65, srv1672408131=29, srv120480784=8, srv1858106515=38, srv470979460=72, srv1845606471=36, srv2018013485=45, srv805578977=86, srv2123211837=53, srv112741249=5, srv2106832016=52, srv2013109571=44, srv1435407348=16, srv818163004=87, srv1054233374=1, srv460209325=70, srv1456399456=17, srv358796505=64, srv1755925672=32, srv795690327=83, srv155623916=23, srv1070885059=3, srv1664319662=28, srv1030653345=0, srv501555138=73, srv313012972=61, srv213867589=54, srv1674170258=30, srv1926640988=41, srv447695486=68, srv804065926=84, srv2141021208=55, srv873378001=91, srv646543227=76, srv1758296038=33, srv2091754783=51, srv904753482=93, srv647891130=78, srv457442417=69, srv1512079295=20, srv357808633=63, srv1138765525=6, srv1314045301=12, srv1058249223=2, srv646784755=77, srv309625420=60, srv1216846790=9, srv1532633762=22, srv1705575887=31, srv1905030220=40, srv46134110=71, srv2048436983=47, srv276821199=57, srv440192557=66, srv916124280=95, srv1929152920=42, srv916971801=96, srv2068231608=50, srv1085821952=4, srv825448834=89, srv805462027=85, srv2053264706=49, srv1330143588=14, srv1601920551=25, srv59194775=74, srv600317870=75, srv880917271=92, srv1456948942=18, srv1822406739=35, srv971920447=98, srv91269851=94, srv1662903068=27, srv2053155197=48, srv675431559=79, srv779373093=81, srv2022037757=46, srv1889134791=39, srv851223456=90, srv1556331307=24, srv1817124160=34, srv313660382=62, srv152571948=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-20 08:55:25,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-20 08:55:25,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-20 08:55:25,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-20 08:55:25,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-20 08:55:25,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-20 08:55:25,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-20 08:55:25,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-20 08:55:25,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-20 08:55:25,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-20 08:55:25,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-20 08:55:25,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-20 08:55:25,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-20 08:55:25,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-20 08:55:25,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-20 08:55:25,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-20 08:55:25,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-20 08:55:25,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-20 08:55:25,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-20 08:55:25,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-20 08:55:25,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-20 08:55:25,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-20 08:55:25,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-20 08:55:25,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-20 08:55:25,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-20 08:55:25,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-20 08:55:25,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-20 08:55:25,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-20 08:55:25,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-20 08:55:25,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-20 08:55:25,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-20 08:55:25,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-20 08:55:25,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-20 08:55:25,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-20 08:55:25,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-20 08:55:25,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-20 08:55:25,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-20 08:55:25,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-20 08:55:25,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-20 08:55:25,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-20 08:55:25,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-20 08:55:25,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-20 08:55:25,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-20 08:55:25,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-20 08:55:25,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-20 08:55:25,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-20 08:55:25,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-20 08:55:25,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-20 08:55:25,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-20 08:55:25,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-20 08:55:25,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-20 08:55:25,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-20 08:55:25,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-20 08:55:25,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-20 08:55:25,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-20 08:55:25,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-20 08:55:25,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-20 08:55:25,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-20 08:55:25,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-20 08:55:25,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-20 08:55:25,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-20 08:55:25,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-20 08:55:25,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-20 08:55:25,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-20 08:55:25,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-20 08:55:25,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-20 08:55:25,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-20 08:55:25,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-20 08:55:25,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-20 08:55:25,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-20 08:55:25,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-20 08:55:25,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-20 08:55:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-20 08:55:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-20 08:55:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-20 08:55:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-20 08:55:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-20 08:55:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-20 08:55:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-20 08:55:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-20 08:55:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-20 08:55:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-20 08:55:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-20 08:55:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-20 08:55:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-20 08:55:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-20 08:55:25,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-20 08:55:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-20 08:55:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-20 08:55:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-20 08:55:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-20 08:55:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-20 08:55:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-20 08:55:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-20 08:55:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-20 08:55:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-20 08:55:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-20 08:55:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-20 08:55:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-20 08:55:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-20 08:55:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-20 08:55:25,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-20 08:55:25,677 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03859937124205303, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0535244890824527, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03859937124205303, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-20 08:55:25,677 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-20 08:55:25,677 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1786440598997908, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03859937124205303, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0535244890824527, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03859937124205303, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17358333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24375, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-20 08:55:25,959 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 414 ms to try 5000 different iterations.  Found a solution that moves 2628 regions; Going from a computed imbalance of 0.1786440598997908 to a new imbalance of 0.016764095181969103. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021603623496951375); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035340257246759196, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.292, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021603623496951375); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01375); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-20 08:55:25,972 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv971920447:78 , srv1070885059:80 , srv1456399456:80 , srv152571948:80 , srv398588155:81 , srv1138765525:82 , srv1320867707:82 , srv1631268622:82 , srv1672408131:82 , srv46134110:82 , srv501555138:82 , srv904753482:82 , srv1030653345:83 , srv1233047650:83 , srv1926640988:83 , srv2053264706:83 , srv760416909:83 , srv805578977:83 , srv880917271:83 , srv1054233374:84 , srv1058249223:84 , srv1085821952:84 , srv1216846790:84 , srv1371794176:84 , srv1855188233:84 , srv313012972:84 , srv447695486:84 , srv457442417:84 , srv795690327:84 , srv825448834:84 , srv942792350:84 , srv1162731849:85 , srv1314045301:85 , srv1435407348:85 , srv1512079295:85 , srv1601920551:85 , srv1705575887:85 , srv1755925672:85 , srv1889134791:85 , srv2053155197:85 , srv2068231608:85 , srv2141021208:85 , srv281652241:85 , srv443351923:85 , srv646784755:85 , srv647891130:85 , srv779373093:85 , srv805462027:85 , srv818163004:85 , srv873378001:85 , srv91269851:85 , srv987495560:85 , srv1222917565:86 , srv1532633762:86 , srv1556331307:86 , srv1674170258:86 , srv1758296038:86 , srv1858106515:86 , srv213867589:86 , srv286794869:86 , srv440192557:86 , srv851223456:86 , srv916124280:86 , srv120480784:87 , srv155623916:87 , srv1662903068:87 , srv1664319662:87 , srv1817124160:87 , srv1822406739:87 , srv1845606471:87 , srv1929152920:87 , srv2018013485:87 , srv2022037757:87 , srv2048436983:87 , srv2091754783:87 , srv276821199:87 , srv309625420:87 , srv313660382:87 , srv357808633:87 , srv358796505:87 , srv646543227:87 , srv804065926:87 , srv820016866:87 , srv1486634619:88 , srv1905030220:88 , srv2013109571:88 , srv600317870:88 , srv916971801:88 , srv1330143588:89 , srv1456948942:89 , srv470979460:89 , srv785404383:89 , srv112741249:90 , srv2106832016:90 , srv267720778:90 , srv59194775:90 , srv2004596344:92 , srv675431559:92 , srv460209325:94 , srv2123211837:533 }
2023-11-20 08:55:25,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-20 08:55:25,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1855188233=37, srv785404383=82, srv760416909=80, srv281652241=58, srv286794869=59, srv1486634619=19, srv1631268622=26, srv1233047650=11, srv1162731849=7, srv267720778=56, srv942792350=97, srv1222917565=10, srv2004596344=43, srv1320867707=13, srv1371794176=15, srv987495560=99, srv443351923=67, srv820016866=88, srv398588155=65, srv1672408131=29, srv120480784=8, srv1858106515=38, srv470979460=72, srv1845606471=36, srv2018013485=45, srv805578977=86, srv2123211837=53, srv112741249=5, srv2106832016=52, srv2013109571=44, srv1435407348=16, srv818163004=87, srv1054233374=1, srv460209325=70, srv1456399456=17, srv358796505=64, srv1755925672=32, srv795690327=83, srv155623916=23, srv1070885059=3, srv1664319662=28, srv1030653345=0, srv501555138=73, srv313012972=61, srv213867589=54, srv1674170258=30, srv1926640988=41, srv447695486=68, srv804065926=84, srv2141021208=55, srv873378001=91, srv646543227=76, srv1758296038=33, srv2091754783=51, srv904753482=93, srv647891130=78, srv457442417=69, srv1512079295=20, srv357808633=63, srv1138765525=6, srv1314045301=12, srv1058249223=2, srv646784755=77, srv309625420=60, srv1216846790=9, srv1532633762=22, srv1705575887=31, srv1905030220=40, srv46134110=71, srv2048436983=47, srv276821199=57, srv440192557=66, srv916124280=95, srv1929152920=42, srv916971801=96, srv2068231608=50, srv1085821952=4, srv825448834=89, srv805462027=85, srv2053264706=49, srv1330143588=14, srv1601920551=25, srv59194775=74, srv600317870=75, srv880917271=92, srv1456948942=18, srv1822406739=35, srv971920447=98, srv91269851=94, srv1662903068=27, srv2053155197=48, srv675431559=79, srv779373093=81, srv2022037757=46, srv1889134791=39, srv851223456=90, srv1556331307=24, srv1817124160=34, srv313660382=62, srv152571948=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-20 08:55:25,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-20 08:55:25,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-20 08:55:25,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-20 08:55:25,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-20 08:55:25,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-20 08:55:25,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-20 08:55:25,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-20 08:55:25,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-20 08:55:25,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-20 08:55:25,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-20 08:55:25,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-20 08:55:25,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-20 08:55:25,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-20 08:55:25,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-20 08:55:25,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-20 08:55:25,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-20 08:55:25,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-20 08:55:25,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-20 08:55:25,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-20 08:55:25,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-20 08:55:25,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-20 08:55:25,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-20 08:55:25,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-20 08:55:25,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-20 08:55:25,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-20 08:55:25,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-20 08:55:25,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-20 08:55:25,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-20 08:55:25,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-20 08:55:25,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-20 08:55:25,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-20 08:55:25,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-20 08:55:25,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-20 08:55:25,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-20 08:55:25,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-20 08:55:25,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-20 08:55:25,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-20 08:55:25,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-20 08:55:25,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-20 08:55:25,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-20 08:55:25,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-20 08:55:25,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-20 08:55:25,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-20 08:55:25,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-20 08:55:25,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-20 08:55:25,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-20 08:55:25,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-20 08:55:25,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-20 08:55:25,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-20 08:55:25,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-20 08:55:25,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-20 08:55:25,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-20 08:55:25,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-20 08:55:25,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-20 08:55:25,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-20 08:55:25,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-20 08:55:25,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-20 08:55:25,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-20 08:55:25,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-20 08:55:25,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-20 08:55:25,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-20 08:55:25,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-20 08:55:25,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-20 08:55:25,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-20 08:55:25,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-20 08:55:25,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-20 08:55:25,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-20 08:55:25,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-20 08:55:25,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-20 08:55:25,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-20 08:55:25,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-20 08:55:25,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-20 08:55:25,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-20 08:55:25,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-20 08:55:25,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-20 08:55:25,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-20 08:55:25,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-20 08:55:25,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-20 08:55:25,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-20 08:55:25,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-20 08:55:25,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-20 08:55:25,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-20 08:55:25,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-20 08:55:25,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-20 08:55:25,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-20 08:55:26,153 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021603623496951375); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035340257246759196, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021603623496951375); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01375); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-20 08:55:26,153 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-20 08:55:26,154 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016745693532662764, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.021603623496951375); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.035340257246759196, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.021603623496951375); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01375); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-20 08:55:26,409 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 380 ms to try 5000 different iterations.  Found a solution that moves 1022 regions; Going from a computed imbalance of 0.016745693532662764 to a new imbalance of 7.762055301177521E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005554433107709862); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009904142134751739); MoveCostFunction : (multiplier=7.0, imbalance=0.11355555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005554433107709862); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-20 08:55:26,413 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv460209325:81 , srv112741249:82 , srv2141021208:82 , srv1058249223:83 , srv1070885059:83 , srv1672408131:83 , srv2004596344:83 , srv2013109571:83 , srv2018013485:83 , srv2106832016:83 , srv600317870:83 , srv675431559:83 , srv880917271:83 , srv916124280:83 , srv1054233374:84 , srv1222917565:84 , srv1314045301:84 , srv1532633762:84 , srv1758296038:84 , srv1822406739:84 , srv2053264706:84 , srv213867589:84 , srv267720778:84 , srv825448834:84 , srv873378001:84 , srv1233047650:85 , srv1330143588:85 , srv1435407348:85 , srv1486634619:85 , srv155623916:85 , srv1601920551:85 , srv1674170258:85 , srv1817124160:85 , srv1889134791:85 , srv1926640988:85 , srv2068231608:85 , srv281652241:85 , srv313012972:85 , srv357808633:85 , srv440192557:85 , srv501555138:85 , srv59194775:85 , srv795690327:85 , srv804065926:85 , srv851223456:85 , srv91269851:85 , srv1085821952:86 , srv1138765525:86 , srv1162731849:86 , srv1216846790:86 , srv1320867707:86 , srv1456399456:86 , srv1456948942:86 , srv152571948:86 , srv1556331307:86 , srv1662903068:86 , srv1705575887:86 , srv1845606471:86 , srv1855188233:86 , srv1858106515:86 , srv1929152920:86 , srv2048436983:86 , srv286794869:86 , srv309625420:86 , srv358796505:86 , srv447695486:86 , srv457442417:86 , srv470979460:86 , srv647891130:86 , srv760416909:86 , srv785404383:86 , srv916971801:86 , srv942792350:86 , srv971920447:86 , srv1030653345:87 , srv120480784:87 , srv1371794176:87 , srv1755925672:87 , srv2022037757:87 , srv2053155197:87 , srv2091754783:87 , srv276821199:87 , srv313660382:87 , srv398588155:87 , srv443351923:87 , srv646543227:87 , srv646784755:87 , srv779373093:87 , srv805462027:87 , srv805578977:87 , srv818163004:87 , srv820016866:87 , srv987495560:87 , srv1512079295:88 , srv1631268622:88 , srv1664319662:88 , srv1905030220:88 , srv46134110:88 , srv904753482:91 , srv2123211837:541 }
2023-11-20 08:55:26,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-20 08:55:26,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1855188233=37, srv785404383=82, srv760416909=80, srv281652241=58, srv286794869=59, srv1486634619=19, srv1631268622=26, srv1233047650=11, srv1162731849=7, srv267720778=56, srv942792350=97, srv1222917565=10, srv2004596344=43, srv1320867707=13, srv1371794176=15, srv987495560=99, srv443351923=67, srv820016866=88, srv398588155=65, srv1672408131=29, srv120480784=8, srv1858106515=38, srv470979460=72, srv1845606471=36, srv2018013485=45, srv805578977=86, srv2123211837=53, srv112741249=5, srv2106832016=52, srv2013109571=44, srv1435407348=16, srv818163004=87, srv1054233374=1, srv460209325=70, srv1456399456=17, srv358796505=64, srv1755925672=32, srv795690327=83, srv155623916=23, srv1070885059=3, srv1664319662=28, srv1030653345=0, srv501555138=73, srv313012972=61, srv213867589=54, srv1674170258=30, srv1926640988=41, srv447695486=68, srv804065926=84, srv2141021208=55, srv873378001=91, srv646543227=76, srv1758296038=33, srv2091754783=51, srv904753482=93, srv647891130=78, srv457442417=69, srv1512079295=20, srv357808633=63, srv1138765525=6, srv1314045301=12, srv1058249223=2, srv646784755=77, srv309625420=60, srv1216846790=9, srv1532633762=22, srv1705575887=31, srv1905030220=40, srv46134110=71, srv2048436983=47, srv276821199=57, srv440192557=66, srv916124280=95, srv1929152920=42, srv916971801=96, srv2068231608=50, srv1085821952=4, srv825448834=89, srv805462027=85, srv2053264706=49, srv1330143588=14, srv1601920551=25, srv59194775=74, srv600317870=75, srv880917271=92, srv1456948942=18, srv1822406739=35, srv971920447=98, srv91269851=94, srv1662903068=27, srv2053155197=48, srv675431559=79, srv779373093=81, srv2022037757=46, srv1889134791=39, srv851223456=90, srv1556331307=24, srv1817124160=34, srv313660382=62, srv152571948=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-20 08:55:26,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-20 08:55:26,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-20 08:55:26,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-20 08:55:26,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-20 08:55:26,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-20 08:55:26,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-20 08:55:26,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-20 08:55:26,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-20 08:55:26,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-20 08:55:26,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-20 08:55:26,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-20 08:55:26,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-20 08:55:26,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-20 08:55:26,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-20 08:55:26,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-20 08:55:26,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-20 08:55:26,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-20 08:55:26,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-20 08:55:26,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-20 08:55:26,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-20 08:55:26,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-20 08:55:26,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-20 08:55:26,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-20 08:55:26,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-20 08:55:26,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-20 08:55:26,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-20 08:55:26,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-20 08:55:26,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-20 08:55:26,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-20 08:55:26,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-20 08:55:26,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-20 08:55:26,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-20 08:55:26,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-20 08:55:26,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-20 08:55:26,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-20 08:55:26,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-20 08:55:26,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-20 08:55:26,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-20 08:55:26,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-20 08:55:26,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-20 08:55:26,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-20 08:55:26,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-20 08:55:26,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-20 08:55:26,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-20 08:55:26,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-20 08:55:26,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-20 08:55:26,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-20 08:55:26,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-20 08:55:26,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-20 08:55:26,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-20 08:55:26,444 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-20 08:55:26,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-20 08:55:26,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-20 08:55:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-20 08:55:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-20 08:55:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-20 08:55:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-20 08:55:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-20 08:55:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-20 08:55:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-20 08:55:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-20 08:55:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-20 08:55:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-20 08:55:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-20 08:55:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-20 08:55:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-20 08:55:26,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-20 08:55:26,611 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005554433107709862); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009904142134751739); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005554433107709862); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-20 08:55:26,612 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-11-20 08:55:26,612 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.690493331652876E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005554433107709862); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.009904142134751739); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005554433107709862); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-11-20 08:55:26,950 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 494 ms to try 5000 different iterations.  Found a solution that moves 222 regions; Going from a computed imbalance of 7.690493331652876E-4 to a new imbalance of 1.860940515396631E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.024666666666666667); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-20 08:55:26,951 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv904753482:81 , srv1631268622:83 , srv1845606471:83 , srv2068231608:83 , srv2091754783:83 , srv987495560:83 , srv1054233374:84 , srv1085821952:84 , srv112741249:84 , srv1216846790:84 , srv1320867707:84 , srv1435407348:84 , srv1456948942:84 , srv1512079295:84 , srv1532633762:84 , srv1601920551:84 , srv1662903068:84 , srv1664319662:84 , srv1817124160:84 , srv1905030220:84 , srv1926640988:84 , srv1929152920:84 , srv2004596344:84 , srv2022037757:84 , srv2048436983:84 , srv267720778:84 , srv313660382:84 , srv357808633:84 , srv358796505:84 , srv440192557:84 , srv443351923:84 , srv457442417:84 , srv46134110:84 , srv470979460:84 , srv501555138:84 , srv59194775:84 , srv600317870:84 , srv646543227:84 , srv646784755:84 , srv647891130:84 , srv779373093:84 , srv805462027:84 , srv805578977:84 , srv825448834:84 , srv916971801:84 , srv971920447:84 , srv1030653345:85 , srv1070885059:85 , srv1138765525:85 , srv1162731849:85 , srv120480784:85 , srv1222917565:85 , srv1233047650:85 , srv1314045301:85 , srv1330143588:85 , srv1371794176:85 , srv1456399456:85 , srv1486634619:85 , srv152571948:85 , srv155623916:85 , srv1556331307:85 , srv1672408131:85 , srv1705575887:85 , srv1755925672:85 , srv1758296038:85 , srv1822406739:85 , srv1855188233:85 , srv1858106515:85 , srv2013109571:85 , srv2018013485:85 , srv2053155197:85 , srv2053264706:85 , srv2106832016:85 , srv213867589:85 , srv2141021208:85 , srv276821199:85 , srv281652241:85 , srv286794869:85 , srv309625420:85 , srv313012972:85 , srv398588155:85 , srv447695486:85 , srv460209325:85 , srv675431559:85 , srv760416909:85 , srv785404383:85 , srv795690327:85 , srv804065926:85 , srv820016866:85 , srv851223456:85 , srv873378001:85 , srv880917271:85 , srv91269851:85 , srv916124280:85 , srv942792350:85 , srv1058249223:86 , srv1674170258:86 , srv1889134791:86 , srv818163004:86 , srv2123211837:635 }
2023-11-20 08:55:26,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-20 08:55:26,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1855188233=37, srv785404383=82, srv760416909=80, srv281652241=58, srv286794869=59, srv1486634619=19, srv1631268622=26, srv1233047650=11, srv1162731849=7, srv267720778=56, srv942792350=97, srv1222917565=10, srv2004596344=43, srv1320867707=13, srv1371794176=15, srv987495560=99, srv443351923=67, srv820016866=88, srv398588155=65, srv1672408131=29, srv120480784=8, srv1858106515=38, srv470979460=72, srv1845606471=36, srv2018013485=45, srv805578977=86, srv2123211837=53, srv112741249=5, srv2106832016=52, srv2013109571=44, srv1435407348=16, srv818163004=87, srv1054233374=1, srv460209325=70, srv1456399456=17, srv358796505=64, srv1755925672=32, srv795690327=83, srv155623916=23, srv1070885059=3, srv1664319662=28, srv1030653345=0, srv501555138=73, srv313012972=61, srv213867589=54, srv1674170258=30, srv1926640988=41, srv447695486=68, srv804065926=84, srv2141021208=55, srv873378001=91, srv646543227=76, srv1758296038=33, srv2091754783=51, srv904753482=93, srv647891130=78, srv457442417=69, srv1512079295=20, srv357808633=63, srv1138765525=6, srv1314045301=12, srv1058249223=2, srv646784755=77, srv309625420=60, srv1216846790=9, srv1532633762=22, srv1705575887=31, srv1905030220=40, srv46134110=71, srv2048436983=47, srv276821199=57, srv440192557=66, srv916124280=95, srv1929152920=42, srv916971801=96, srv2068231608=50, srv1085821952=4, srv825448834=89, srv805462027=85, srv2053264706=49, srv1330143588=14, srv1601920551=25, srv59194775=74, srv600317870=75, srv880917271=92, srv1456948942=18, srv1822406739=35, srv971920447=98, srv91269851=94, srv1662903068=27, srv2053155197=48, srv675431559=79, srv779373093=81, srv2022037757=46, srv1889134791=39, srv851223456=90, srv1556331307=24, srv1817124160=34, srv313660382=62, srv152571948=21} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-20 08:55:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-20 08:55:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-20 08:55:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-20 08:55:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-20 08:55:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-20 08:55:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-20 08:55:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-20 08:55:26,974 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-20 08:55:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-20 08:55:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-20 08:55:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-20 08:55:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-20 08:55:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-20 08:55:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-20 08:55:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-20 08:55:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-20 08:55:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-20 08:55:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-20 08:55:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-20 08:55:26,975 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-20 08:55:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-20 08:55:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-20 08:55:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-20 08:55:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-20 08:55:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-20 08:55:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-20 08:55:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-20 08:55:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-20 08:55:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-20 08:55:26,976 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-20 08:55:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-11-20 08:55:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-11-20 08:55:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-11-20 08:55:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-11-20 08:55:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-11-20 08:55:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-11-20 08:55:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-11-20 08:55:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-11-20 08:55:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-11-20 08:55:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-11-20 08:55:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-11-20 08:55:26,977 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-11-20 08:55:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-11-20 08:55:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-11-20 08:55:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-11-20 08:55:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-11-20 08:55:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-11-20 08:55:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-11-20 08:55:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-11-20 08:55:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-11-20 08:55:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-11-20 08:55:26,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-11-20 08:55:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-11-20 08:55:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-11-20 08:55:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-11-20 08:55:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-11-20 08:55:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-11-20 08:55:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-11-20 08:55:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-11-20 08:55:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-11-20 08:55:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-11-20 08:55:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-11-20 08:55:26,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-11-20 08:55:26,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-11-20 08:55:26,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-11-20 08:55:26,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-11-20 08:55:26,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-11-20 08:55:26,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-11-20 08:55:26,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-11-20 08:55:26,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-11-20 08:55:26,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-11-20 08:55:26,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-11-20 08:55:26,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-11-20 08:55:26,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-11-20 08:55:26,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-11-20 08:55:26,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-11-20 08:55:26,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-11-20 08:55:26,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-11-20 08:55:26,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-11-20 08:55:26,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-11-20 08:55:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-11-20 08:55:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-11-20 08:55:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-11-20 08:55:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-11-20 08:55:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-11-20 08:55:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-11-20 08:55:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-11-20 08:55:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-11-20 08:55:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-11-20 08:55:26,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-11-20 08:55:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-11-20 08:55:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-11-20 08:55:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-11-20 08:55:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-11-20 08:55:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-11-20 08:55:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-11-20 08:55:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-11-20 08:55:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-11-20 08:55:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-11-20 08:55:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-11-20 08:55:26,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-11-20 08:55:27,149 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.7054927929458383E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-20 08:55:27,150 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv904753482:81 , srv1631268622:83 , srv1845606471:83 , srv2068231608:83 , srv2091754783:83 , srv987495560:83 , srv1054233374:84 , srv1085821952:84 , srv112741249:84 , srv1216846790:84 , srv1320867707:84 , srv1435407348:84 , srv1456948942:84 , srv1512079295:84 , srv1532633762:84 , srv1601920551:84 , srv1662903068:84 , srv1664319662:84 , srv1817124160:84 , srv1905030220:84 , srv1926640988:84 , srv1929152920:84 , srv2004596344:84 , srv2022037757:84 , srv2048436983:84 , srv267720778:84 , srv313660382:84 , srv357808633:84 , srv358796505:84 , srv440192557:84 , srv443351923:84 , srv457442417:84 , srv46134110:84 , srv470979460:84 , srv501555138:84 , srv59194775:84 , srv600317870:84 , srv646543227:84 , srv646784755:84 , srv647891130:84 , srv779373093:84 , srv805462027:84 , srv805578977:84 , srv825448834:84 , srv916971801:84 , srv971920447:84 , srv1030653345:85 , srv1070885059:85 , srv1138765525:85 , srv1162731849:85 , srv120480784:85 , srv1222917565:85 , srv1233047650:85 , srv1314045301:85 , srv1330143588:85 , srv1371794176:85 , srv1456399456:85 , srv1486634619:85 , srv152571948:85 , srv155623916:85 , srv1556331307:85 , srv1672408131:85 , srv1705575887:85 , srv1755925672:85 , srv1758296038:85 , srv1822406739:85 , srv1855188233:85 , srv1858106515:85 , srv2013109571:85 , srv2018013485:85 , srv2053155197:85 , srv2053264706:85 , srv2106832016:85 , srv213867589:85 , srv2141021208:85 , srv276821199:85 , srv281652241:85 , srv286794869:85 , srv309625420:85 , srv313012972:85 , srv398588155:85 , srv447695486:85 , srv460209325:85 , srv675431559:85 , srv760416909:85 , srv785404383:85 , srv795690327:85 , srv804065926:85 , srv820016866:85 , srv851223456:85 , srv873378001:85 , srv880917271:85 , srv91269851:85 , srv916124280:85 , srv942792350:85 , srv1058249223:86 , srv1674170258:86 , srv1889134791:86 , srv818163004:86 , srv2123211837:635 }
2023-11-20 08:55:27,183 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=117 (was 93) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=15163 (was 15648)
2023-11-20 08:55:27,190 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=117, ProcessCount=168, AvailableMemoryMB=15163
2023-11-20 08:55:27,190 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-11-20 08:55:27,190 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-11-20 08:55:27,202 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv102648910:84 , srv1362062373:84 , srv1426971821:84 , srv1528433392:84 , srv1708004826:84 , srv1808342680:84 , srv1809235430:84 , srv1946774733:84 , srv1955200452:84 , srv1981874586:84 , srv236543705:84 , srv275949436:84 , srv371400231:84 , srv371628204:84 , srv425962695:84 , srv439419842:84 , srv521210829:84 , srv521984171:84 , srv528278498:84 , srv530981080:84 , srv564530179:84 , srv614825643:84 , srv637780084:84 , srv679686703:84 , srv825636585:84 , srv830925201:84 , srv899190825:84 , srv905667013:84 , srv998197525:84 , srv528503981:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-11-20 08:55:27,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-20 08:55:27,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv899190825=27, srv425962695=14, srv1528433392=3, srv1946774733=7, srv439419842=15, srv530981080=20, srv371400231=12, srv998197525=29, srv1955200452=8, srv564530179=21, srv102648910=0, srv679686703=24, srv371628204=13, srv1426971821=2, srv637780084=23, srv236543705=10, srv614825643=22, srv830925201=26, srv275949436=11, srv528278498=18, srv528503981=19, srv1708004826=4, srv521984171=17, srv905667013=28, srv825636585=25, srv1981874586=9, srv1362062373=1, srv1808342680=5, srv1809235430=6, srv521210829=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-20 08:55:27,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-20 08:55:27,219 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-20 08:55:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-20 08:55:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-20 08:55:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-20 08:55:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-20 08:55:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-20 08:55:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-20 08:55:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-20 08:55:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-20 08:55:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-20 08:55:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-20 08:55:27,220 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-20 08:55:27,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-20 08:55:27,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-20 08:55:27,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-20 08:55:27,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-20 08:55:27,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-20 08:55:27,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-20 08:55:27,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-20 08:55:27,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-20 08:55:27,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-20 08:55:27,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-20 08:55:27,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-20 08:55:27,221 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-20 08:55:27,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-20 08:55:27,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-20 08:55:27,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-20 08:55:27,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-20 08:55:27,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-20 08:55:27,222 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-20 08:55:27,241 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1066993990702705, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-20 08:55:27,241 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912529249586695, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1066993990702705, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-11-20 08:57:27,226 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5700269 different iterations.  Found a solution that moves 1856 regions; Going from a computed imbalance of 0.9912529249586695 to a new imbalance of 4.3319965896196796E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6874074074074074, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-20 08:57:27,233 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv102648910:90 , srv1362062373:90 , srv1426971821:90 , srv1528433392:90 , srv1708004826:90 , srv1808342680:90 , srv1809235430:90 , srv1946774733:90 , srv1955200452:90 , srv1981874586:90 , srv236543705:90 , srv275949436:90 , srv371400231:90 , srv371628204:90 , srv425962695:90 , srv439419842:90 , srv521210829:90 , srv521984171:90 , srv528278498:90 , srv528503981:90 , srv530981080:90 , srv564530179:90 , srv614825643:90 , srv637780084:90 , srv679686703:90 , srv825636585:90 , srv830925201:90 , srv899190825:90 , srv905667013:90 , srv998197525:90 }
2023-11-20 08:57:27,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-11-20 08:57:27,235 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv899190825=27, srv425962695=14, srv1528433392=3, srv1946774733=7, srv439419842=15, srv530981080=20, srv371400231=12, srv998197525=29, srv1955200452=8, srv564530179=21, srv102648910=0, srv679686703=24, srv371628204=13, srv1426971821=2, srv637780084=23, srv236543705=10, srv614825643=22, srv830925201=26, srv275949436=11, srv528278498=18, srv528503981=19, srv1708004826=4, srv521984171=17, srv905667013=28, srv825636585=25, srv1981874586=9, srv1362062373=1, srv1808342680=5, srv1809235430=6, srv521210829=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-11-20 08:57:27,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-11-20 08:57:27,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-11-20 08:57:27,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-11-20 08:57:27,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-11-20 08:57:27,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-11-20 08:57:27,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-11-20 08:57:27,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-11-20 08:57:27,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-11-20 08:57:27,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-11-20 08:57:27,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-11-20 08:57:27,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-11-20 08:57:27,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-11-20 08:57:27,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-11-20 08:57:27,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-11-20 08:57:27,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-11-20 08:57:27,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-11-20 08:57:27,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-11-20 08:57:27,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-11-20 08:57:27,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-11-20 08:57:27,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-11-20 08:57:27,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-11-20 08:57:27,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-11-20 08:57:27,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-11-20 08:57:27,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-11-20 08:57:27,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-11-20 08:57:27,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-11-20 08:57:27,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-11-20 08:57:27,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-11-20 08:57:27,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-11-20 08:57:27,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-11-20 08:57:27,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-11-20 08:57:27,261 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-11-20 08:57:27,261 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv102648910:90 , srv1362062373:90 , srv1426971821:90 , srv1528433392:90 , srv1708004826:90 , srv1808342680:90 , srv1809235430:90 , srv1946774733:90 , srv1955200452:90 , srv1981874586:90 , srv236543705:90 , srv275949436:90 , srv371400231:90 , srv371628204:90 , srv425962695:90 , srv439419842:90 , srv521210829:90 , srv521984171:90 , srv528278498:90 , srv528503981:90 , srv530981080:90 , srv564530179:90 , srv614825643:90 , srv637780084:90 , srv679686703:90 , srv825636585:90 , srv830925201:90 , srv899190825:90 , srv905667013:90 , srv998197525:90 }
2023-11-20 08:57:27,275 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=200 (was 117) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=15396 (was 15163) - AvailableMemoryMB LEAK? -