2023-03-06 09:56:59,466 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-06 09:56:59,817 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-06 09:56:59,948 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-06 09:57:00,016 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-06 09:57:00,023 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-06 09:57:00,036 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=737, ProcessCount=181, AvailableMemoryMB=7090
2023-03-06 09:57:00,038 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-06 09:57:00,038 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-06 09:57:00,194 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000422246:84 , srv1003634171:84 , srv1019689032:84 , srv1028844665:84 , srv103040898:84 , srv1036833636:84 , srv1056037271:84 , srv1060846971:84 , srv1108914261:84 , srv1119273761:84 , srv1161895927:84 , srv1166863823:84 , srv120580797:84 , srv1205938727:84 , srv1232954782:84 , srv1251933708:84 , srv1264406980:84 , srv1285789177:84 , srv1297969997:84 , srv1325018111:84 , srv1337364081:84 , srv1338914418:84 , srv1344909140:84 , srv1351263768:84 , srv1425804394:84 , srv1454456:84 , srv1476658600:84 , srv1485374761:84 , srv1504038393:84 , srv1557090697:84 , srv1566343790:84 , srv1608894804:84 , srv1655433505:84 , srv1671705647:84 , srv1677042635:84 , srv1702701761:84 , srv1717713807:84 , srv1729959962:84 , srv1755914834:84 , srv1794868146:84 , srv1825850676:84 , srv1830469763:84 , srv1942630762:84 , srv1943153373:84 , srv1948618655:84 , srv1949975054:84 , srv1954432928:84 , srv1971278706:84 , srv1972603170:84 , srv1981560900:84 , srv19859932:84 , srv1989810761:84 , srv201333103:84 , srv2021112311:84 , srv2041973834:84 , srv2064267976:84 , srv2073761099:84 , srv2138651010:84 , srv218671853:84 , srv232604783:84 , srv249651420:84 , srv263531896:84 , srv323057992:84 , srv333840381:84 , srv34769195:84 , srv355058911:84 , srv355706026:84 , srv368747626:84 , srv388175903:84 , srv424357571:84 , srv434819608:84 , srv446095969:84 , srv472645754:84 , srv518668489:84 , srv523128973:84 , srv539887239:84 , srv567692858:84 , srv568909358:84 , srv578408332:84 , srv604382155:84 , srv635040161:84 , srv639008775:84 , srv648604316:84 , srv667751141:84 , srv711554003:84 , srv719411698:84 , srv725621880:84 , srv75144734:84 , srv752622305:84 , srv813086408:84 , srv818589027:84 , srv819754298:84 , srv864665878:84 , srv913800862:84 , srv932573851:84 , srv955526783:84 , srv962482486:84 , srv969020705:84 , srv974907539:84 , srv1592475274:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-06 09:57:00,206 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-06 09:57:00,215 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1019689032=2, srv1981560900=50, srv539887239=76, srv1003634171=1, srv2064267976=56, srv1485374761=27, srv864665878=93, srv955526783=96, srv1943153373=44, srv2041973834=55, srv1671705647=34, srv1060846971=7, srv1108914261=8, srv232604783=60, srv523128973=75, srv1655433505=33, srv1285789177=17, srv1608894804=32, srv604382155=80, srv962482486=97, srv120580797=12, srv1119273761=9, srv1954432928=47, srv1166863823=11, srv424357571=70, srv818589027=91, srv1717713807=37, srv1971278706=48, srv667751141=84, srv719411698=86, srv1504038393=28, srv1592475274=31, srv1425804394=24, srv1337364081=20, srv1297969997=18, srv1677042635=35, srv1702701761=36, srv355058911=66, srv974907539=99, srv1557090697=29, srv752622305=89, srv1351263768=23, srv323057992=63, srv913800862=94, srv434819608=71, srv103040898=4, srv1948618655=45, srv75144734=88, srv1264406980=16, srv1755914834=39, srv201333103=53, srv1972603170=49, srv711554003=85, srv1989810761=52, srv355706026=67, srv1344909140=22, srv388175903=69, srv813086408=90, srv639008775=82, srv2073761099=57, srv518668489=74, srv34769195=65, srv263531896=62, srv648604316=83, srv1000422246=0, srv1949975054=46, srv1454456=25, srv2021112311=54, srv578408332=79, srv1232954782=14, srv932573851=95, srv568909358=78, srv249651420=61, srv1476658600=26, srv567692858=77, srv969020705=98, srv1566343790=30, srv368747626=68, srv635040161=81, srv1251933708=15, srv446095969=72, srv1161895927=10, srv1325018111=19, srv1036833636=5, srv1825850676=41, srv1794868146=40, srv218671853=59, srv1205938727=13, srv1338914418=21, srv1830469763=42, srv333840381=64, srv1056037271=6, srv2138651010=58, srv725621880=87, srv472645754=73, srv1729959962=38, srv1942630762=43, srv819754298=92, srv19859932=51, srv1028844665=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-06 09:57:00,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-06 09:57:00,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-06 09:57:00,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-06 09:57:00,352 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-06 09:57:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-06 09:57:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-06 09:57:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-06 09:57:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-06 09:57:00,353 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-06 09:57:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-06 09:57:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-06 09:57:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-06 09:57:00,354 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-06 09:57:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-06 09:57:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-06 09:57:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-06 09:57:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-06 09:57:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-06 09:57:00,355 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-06 09:57:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-06 09:57:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-06 09:57:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-06 09:57:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-06 09:57:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-06 09:57:00,356 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-06 09:57:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-06 09:57:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-06 09:57:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-06 09:57:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-06 09:57:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-06 09:57:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-06 09:57:00,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-06 09:57:00,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-06 09:57:00,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-06 09:57:00,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-06 09:57:00,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-06 09:57:00,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-06 09:57:00,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-06 09:57:00,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-06 09:57:00,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-06 09:57:00,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-06 09:57:00,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-06 09:57:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-06 09:57:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-06 09:57:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-06 09:57:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-06 09:57:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-06 09:57:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-06 09:57:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-06 09:57:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-06 09:57:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-06 09:57:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-06 09:57:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-06 09:57:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-06 09:57:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-06 09:57:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-06 09:57:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-06 09:57:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-06 09:57:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-06 09:57:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-06 09:57:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-06 09:57:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-06 09:57:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-06 09:57:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-06 09:57:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-06 09:57:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-06 09:57:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-06 09:57:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-06 09:57:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-06 09:57:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-06 09:57:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-06 09:57:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-06 09:57:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-06 09:57:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-06 09:57:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-06 09:57:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-06 09:57:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-06 09:57:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-06 09:57:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-06 09:57:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-06 09:57:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-06 09:57:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-06 09:57:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-06 09:57:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-06 09:57:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-06 09:57:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-06 09:57:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-06 09:57:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-06 09:57:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-06 09:57:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-06 09:57:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-06 09:57:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-06 09:57:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-06 09:57:00,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-06 09:57:00,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-06 09:57:00,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-06 09:57:00,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-06 09:57:00,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-06 09:57:00,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-06 09:57:00,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-06 09:57:00,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-06 09:57:00,617 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-06 09:57:00,618 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-06 09:57:00,618 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-06 09:57:00,931 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 518 ms to try 5000 different iterations.  Found a solution that moves 4730 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17429510575859344. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03857157702808145, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05839222616340931, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5255555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03857157702808145, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16925, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23816666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-06 09:57:00,951 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv567692858:81 , srv1949975054:82 , srv518668489:82 , srv1056037271:83 , srv1454456:83 , srv1485374761:83 , srv1755914834:83 , srv752622305:83 , srv813086408:83 , srv1000422246:84 , srv1028844665:84 , srv1060846971:84 , srv1729959962:84 , srv1942630762:84 , srv2064267976:84 , srv2138651010:84 , srv249651420:84 , srv725621880:84 , srv1161895927:85 , srv1285789177:85 , srv1717713807:85 , srv1971278706:85 , srv1972603170:85 , srv2021112311:85 , srv218671853:85 , srv446095969:85 , srv539887239:85 , srv648604316:85 , srv1019689032:86 , srv1036833636:86 , srv1205938727:86 , srv1232954782:86 , srv1504038393:86 , srv1566343790:86 , srv1655433505:86 , srv1671705647:86 , srv1825850676:86 , srv1943153373:86 , srv1954432928:86 , srv1989810761:86 , srv323057992:86 , srv333840381:86 , srv355058911:86 , srv472645754:86 , srv523128973:86 , srv711554003:86 , srv932573851:86 , srv974907539:86 , srv1119273761:87 , srv1251933708:87 , srv1325018111:87 , srv1344909140:87 , srv1557090697:87 , srv1794868146:87 , srv1830469763:87 , srv1948618655:87 , srv232604783:87 , srv355706026:87 , srv368747626:87 , srv388175903:87 , srv424357571:87 , srv434819608:87 , srv578408332:87 , srv639008775:87 , srv667751141:87 , srv719411698:87 , srv818589027:87 , srv819754298:87 , srv969020705:87 , srv103040898:88 , srv120580797:88 , srv1264406980:88 , srv1338914418:88 , srv1351263768:88 , srv1702701761:88 , srv2041973834:88 , srv263531896:88 , srv34769195:88 , srv568909358:88 , srv604382155:88 , srv75144734:88 , srv913800862:88 , srv1166863823:89 , srv1337364081:89 , srv1425804394:89 , srv1981560900:89 , srv19859932:89 , srv201333103:89 , srv2073761099:89 , srv635040161:89 , srv1003634171:90 , srv1297969997:90 , srv1608894804:90 , srv1677042635:90 , srv864665878:90 , srv955526783:90 , srv1476658600:91 , srv1108914261:92 , srv962482486:92 , srv1592475274:433 }
2023-03-06 09:57:00,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-06 09:57:00,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1019689032=2, srv1981560900=50, srv539887239=76, srv1003634171=1, srv2064267976=56, srv1485374761=27, srv864665878=93, srv955526783=96, srv1943153373=44, srv2041973834=55, srv1671705647=34, srv1060846971=7, srv1108914261=8, srv232604783=60, srv523128973=75, srv1655433505=33, srv1285789177=17, srv1608894804=32, srv604382155=80, srv962482486=97, srv120580797=12, srv1119273761=9, srv1954432928=47, srv1166863823=11, srv424357571=70, srv818589027=91, srv1717713807=37, srv1971278706=48, srv667751141=84, srv719411698=86, srv1504038393=28, srv1592475274=31, srv1425804394=24, srv1337364081=20, srv1297969997=18, srv1677042635=35, srv1702701761=36, srv355058911=66, srv974907539=99, srv1557090697=29, srv752622305=89, srv1351263768=23, srv323057992=63, srv913800862=94, srv434819608=71, srv103040898=4, srv1948618655=45, srv75144734=88, srv1264406980=16, srv1755914834=39, srv201333103=53, srv1972603170=49, srv711554003=85, srv1989810761=52, srv355706026=67, srv1344909140=22, srv388175903=69, srv813086408=90, srv639008775=82, srv2073761099=57, srv518668489=74, srv34769195=65, srv263531896=62, srv648604316=83, srv1000422246=0, srv1949975054=46, srv1454456=25, srv2021112311=54, srv578408332=79, srv1232954782=14, srv932573851=95, srv568909358=78, srv249651420=61, srv1476658600=26, srv567692858=77, srv969020705=98, srv1566343790=30, srv368747626=68, srv635040161=81, srv1251933708=15, srv446095969=72, srv1161895927=10, srv1325018111=19, srv1036833636=5, srv1825850676=41, srv1794868146=40, srv218671853=59, srv1205938727=13, srv1338914418=21, srv1830469763=42, srv333840381=64, srv1056037271=6, srv2138651010=58, srv725621880=87, srv472645754=73, srv1729959962=38, srv1942630762=43, srv819754298=92, srv19859932=51, srv1028844665=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-06 09:57:00,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-06 09:57:00,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-06 09:57:00,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-06 09:57:00,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-06 09:57:00,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-06 09:57:00,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-06 09:57:00,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-06 09:57:00,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-06 09:57:00,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-06 09:57:00,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-06 09:57:00,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-06 09:57:00,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-06 09:57:00,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-06 09:57:00,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-06 09:57:00,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-06 09:57:00,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-06 09:57:00,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-06 09:57:00,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-06 09:57:00,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-06 09:57:00,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-06 09:57:00,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-06 09:57:00,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-06 09:57:00,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-06 09:57:00,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-06 09:57:00,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-06 09:57:00,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-06 09:57:00,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-06 09:57:00,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-06 09:57:00,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-06 09:57:00,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-06 09:57:00,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-06 09:57:00,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-06 09:57:00,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-06 09:57:00,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-06 09:57:00,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-06 09:57:00,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-06 09:57:00,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-06 09:57:00,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-06 09:57:00,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-06 09:57:00,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-06 09:57:00,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-06 09:57:00,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-06 09:57:00,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-06 09:57:00,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-06 09:57:00,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-06 09:57:00,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-06 09:57:00,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-06 09:57:00,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-06 09:57:00,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-06 09:57:00,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-06 09:57:00,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-06 09:57:00,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-06 09:57:00,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-06 09:57:00,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-06 09:57:00,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-06 09:57:00,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-06 09:57:00,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-06 09:57:00,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-06 09:57:00,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-06 09:57:00,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-06 09:57:00,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-06 09:57:00,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-06 09:57:00,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-06 09:57:00,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-06 09:57:00,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-06 09:57:00,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-06 09:57:00,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-06 09:57:00,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-06 09:57:00,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-06 09:57:00,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-06 09:57:00,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-06 09:57:00,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-06 09:57:00,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-06 09:57:00,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-06 09:57:00,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-06 09:57:00,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-06 09:57:00,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-06 09:57:00,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-06 09:57:00,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-06 09:57:00,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-06 09:57:00,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-06 09:57:00,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-06 09:57:00,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-06 09:57:00,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-06 09:57:00,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-06 09:57:00,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-06 09:57:00,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-06 09:57:00,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-06 09:57:00,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-06 09:57:00,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-06 09:57:00,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-06 09:57:00,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-06 09:57:00,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-06 09:57:00,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-06 09:57:00,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-06 09:57:00,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-06 09:57:00,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-06 09:57:00,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-06 09:57:00,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-06 09:57:00,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-06 09:57:00,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-06 09:57:01,156 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03857157702808145, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05839222616340931, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03857157702808145, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16925, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23816666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-06 09:57:01,156 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-06 09:57:01,157 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1742619855907019, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03857157702808145, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05839222616340931, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03857157702808145, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.16925, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23816666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-06 09:57:01,416 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 393 ms to try 5000 different iterations.  Found a solution that moves 2630 regions; Going from a computed imbalance of 0.1742619855907019 to a new imbalance of 0.015024660275888719. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02002181127667656); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03394059641929438, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2922222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02002181127667656); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-06 09:57:01,428 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1108914261:79 , srv962482486:79 , srv1003634171:80 , srv1297969997:82 , srv1608894804:82 , srv2073761099:82 , srv955526783:82 , srv1476658600:83 , srv1702701761:83 , srv19859932:83 , srv263531896:83 , srv34769195:83 , srv424357571:83 , srv635040161:83 , srv75144734:83 , srv864665878:83 , srv103040898:84 , srv1166863823:84 , srv1264406980:84 , srv1337364081:84 , srv1351263768:84 , srv1425804394:84 , srv1504038393:84 , srv1557090697:84 , srv1981560900:84 , srv2041973834:84 , srv355706026:84 , srv568909358:84 , srv578408332:84 , srv819754298:84 , srv974907539:84 , srv1119273761:85 , srv120580797:85 , srv1232954782:85 , srv1251933708:85 , srv1344909140:85 , srv1830469763:85 , srv1948618655:85 , srv201333103:85 , srv232604783:85 , srv323057992:85 , srv368747626:85 , srv434819608:85 , srv523128973:85 , srv604382155:85 , srv667751141:85 , srv913800862:85 , srv932573851:85 , srv969020705:85 , srv1019689032:86 , srv1036833636:86 , srv1161895927:86 , srv1205938727:86 , srv1566343790:86 , srv1677042635:86 , srv1717713807:86 , srv1794868146:86 , srv1943153373:86 , srv2064267976:86 , srv218671853:86 , srv472645754:86 , srv539887239:86 , srv711554003:86 , srv719411698:86 , srv818589027:86 , srv1000422246:87 , srv1285789177:87 , srv1338914418:87 , srv1655433505:87 , srv1671705647:87 , srv1942630762:87 , srv1954432928:87 , srv1972603170:87 , srv2138651010:87 , srv249651420:87 , srv333840381:87 , srv355058911:87 , srv388175903:87 , srv446095969:87 , srv639008775:87 , srv648604316:87 , srv1060846971:88 , srv1325018111:88 , srv1825850676:88 , srv1989810761:88 , srv2021112311:88 , srv725621880:88 , srv752622305:88 , srv1028844665:89 , srv1454456:89 , srv1755914834:89 , srv1971278706:89 , srv1485374761:90 , srv1729959962:90 , srv1056037271:91 , srv518668489:91 , srv813086408:91 , srv1949975054:92 , srv567692858:93 , srv1592475274:519 }
2023-03-06 09:57:01,434 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-06 09:57:01,435 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1019689032=2, srv1981560900=50, srv539887239=76, srv1003634171=1, srv2064267976=56, srv1485374761=27, srv864665878=93, srv955526783=96, srv1943153373=44, srv2041973834=55, srv1671705647=34, srv1060846971=7, srv1108914261=8, srv232604783=60, srv523128973=75, srv1655433505=33, srv1285789177=17, srv1608894804=32, srv604382155=80, srv962482486=97, srv120580797=12, srv1119273761=9, srv1954432928=47, srv1166863823=11, srv424357571=70, srv818589027=91, srv1717713807=37, srv1971278706=48, srv667751141=84, srv719411698=86, srv1504038393=28, srv1592475274=31, srv1425804394=24, srv1337364081=20, srv1297969997=18, srv1677042635=35, srv1702701761=36, srv355058911=66, srv974907539=99, srv1557090697=29, srv752622305=89, srv1351263768=23, srv323057992=63, srv913800862=94, srv434819608=71, srv103040898=4, srv1948618655=45, srv75144734=88, srv1264406980=16, srv1755914834=39, srv201333103=53, srv1972603170=49, srv711554003=85, srv1989810761=52, srv355706026=67, srv1344909140=22, srv388175903=69, srv813086408=90, srv639008775=82, srv2073761099=57, srv518668489=74, srv34769195=65, srv263531896=62, srv648604316=83, srv1000422246=0, srv1949975054=46, srv1454456=25, srv2021112311=54, srv578408332=79, srv1232954782=14, srv932573851=95, srv568909358=78, srv249651420=61, srv1476658600=26, srv567692858=77, srv969020705=98, srv1566343790=30, srv368747626=68, srv635040161=81, srv1251933708=15, srv446095969=72, srv1161895927=10, srv1325018111=19, srv1036833636=5, srv1825850676=41, srv1794868146=40, srv218671853=59, srv1205938727=13, srv1338914418=21, srv1830469763=42, srv333840381=64, srv1056037271=6, srv2138651010=58, srv725621880=87, srv472645754=73, srv1729959962=38, srv1942630762=43, srv819754298=92, srv19859932=51, srv1028844665=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-06 09:57:01,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-06 09:57:01,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-06 09:57:01,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-06 09:57:01,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-06 09:57:01,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-06 09:57:01,449 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-06 09:57:01,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-06 09:57:01,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-06 09:57:01,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-06 09:57:01,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-06 09:57:01,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-06 09:57:01,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-06 09:57:01,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-06 09:57:01,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-06 09:57:01,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-06 09:57:01,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-06 09:57:01,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-06 09:57:01,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-06 09:57:01,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-06 09:57:01,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-06 09:57:01,450 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-06 09:57:01,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-06 09:57:01,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-06 09:57:01,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-06 09:57:01,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-06 09:57:01,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-06 09:57:01,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-06 09:57:01,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-06 09:57:01,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-06 09:57:01,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-06 09:57:01,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-06 09:57:01,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-06 09:57:01,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-06 09:57:01,451 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-06 09:57:01,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-06 09:57:01,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-06 09:57:01,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-06 09:57:01,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-06 09:57:01,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-06 09:57:01,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-06 09:57:01,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-06 09:57:01,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-06 09:57:01,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-06 09:57:01,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-06 09:57:01,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-06 09:57:01,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-06 09:57:01,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-06 09:57:01,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-06 09:57:01,452 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-06 09:57:01,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-06 09:57:01,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-06 09:57:01,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-06 09:57:01,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-06 09:57:01,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-06 09:57:01,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-06 09:57:01,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-06 09:57:01,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-06 09:57:01,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-06 09:57:01,453 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-06 09:57:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-06 09:57:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-06 09:57:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-06 09:57:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-06 09:57:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-06 09:57:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-06 09:57:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-06 09:57:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-06 09:57:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-06 09:57:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-06 09:57:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-06 09:57:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-06 09:57:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-06 09:57:01,454 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-06 09:57:01,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-06 09:57:01,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-06 09:57:01,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-06 09:57:01,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-06 09:57:01,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-06 09:57:01,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-06 09:57:01,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-06 09:57:01,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-06 09:57:01,455 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-06 09:57:01,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-06 09:57:01,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-06 09:57:01,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-06 09:57:01,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-06 09:57:01,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-06 09:57:01,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-06 09:57:01,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-06 09:57:01,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-06 09:57:01,456 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-06 09:57:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-06 09:57:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-06 09:57:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-06 09:57:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-06 09:57:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-06 09:57:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-06 09:57:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-06 09:57:01,457 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-06 09:57:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-06 09:57:01,458 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-06 09:57:01,582 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02002181127667656); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03394059641929438, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02002181127667656); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-06 09:57:01,583 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-06 09:57:01,583 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015006244622283062, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02002181127667656); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03394059641929438, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02002181127667656); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011833333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-06 09:57:01,828 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 355 ms to try 5000 different iterations.  Found a solution that moves 953 regions; Going from a computed imbalance of 0.015006244622283062 to a new imbalance of 5.22841296742321E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00767691535093358); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.013121208233216896); MoveCostFunction : (multiplier=7.0, imbalance=0.1058888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00767691535093358); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-06 09:57:01,830 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1056037271:82 , srv1325018111:82 , srv1338914418:82 , srv1989810761:82 , srv1677042635:83 , srv1729959962:83 , srv1794868146:83 , srv1825850676:83 , srv1949975054:83 , srv1971278706:83 , srv388175903:83 , srv567692858:83 , srv639008775:83 , srv719411698:83 , srv813086408:83 , srv913800862:83 , srv1297969997:84 , srv1337364081:84 , srv1476658600:84 , srv1485374761:84 , srv1954432928:84 , srv1981560900:84 , srv201333103:84 , srv333840381:84 , srv518668489:84 , srv604382155:84 , srv818589027:84 , srv864665878:84 , srv1019689032:85 , srv1028844665:85 , srv103040898:85 , srv1108914261:85 , srv1119273761:85 , srv1166863823:85 , srv120580797:85 , srv1351263768:85 , srv1425804394:85 , srv1557090697:85 , srv1566343790:85 , srv1608894804:85 , srv1655433505:85 , srv1671705647:85 , srv1830469763:85 , srv1943153373:85 , srv1972603170:85 , srv19859932:85 , srv2021112311:85 , srv2041973834:85 , srv232604783:85 , srv355058911:85 , srv368747626:85 , srv434819608:85 , srv539887239:85 , srv635040161:85 , srv667751141:85 , srv75144734:85 , srv752622305:85 , srv969020705:85 , srv1000422246:86 , srv1036833636:86 , srv1060846971:86 , srv1161895927:86 , srv1205938727:86 , srv1232954782:86 , srv1251933708:86 , srv1264406980:86 , srv1285789177:86 , srv1344909140:86 , srv1717713807:86 , srv1755914834:86 , srv1942630762:86 , srv1948618655:86 , srv2073761099:86 , srv2138651010:86 , srv218671853:86 , srv263531896:86 , srv34769195:86 , srv355706026:86 , srv446095969:86 , srv472645754:86 , srv568909358:86 , srv648604316:86 , srv711554003:86 , srv725621880:86 , srv819754298:86 , srv932573851:86 , srv962482486:86 , srv1454456:87 , srv1702701761:87 , srv249651420:87 , srv323057992:87 , srv424357571:87 , srv955526783:87 , srv974907539:87 , srv1003634171:88 , srv1504038393:88 , srv523128973:88 , srv578408332:88 , srv2064267976:89 , srv1592475274:574 }
2023-03-06 09:57:01,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-06 09:57:01,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1019689032=2, srv1981560900=50, srv539887239=76, srv1003634171=1, srv2064267976=56, srv1485374761=27, srv864665878=93, srv955526783=96, srv1943153373=44, srv2041973834=55, srv1671705647=34, srv1060846971=7, srv1108914261=8, srv232604783=60, srv523128973=75, srv1655433505=33, srv1285789177=17, srv1608894804=32, srv604382155=80, srv962482486=97, srv120580797=12, srv1119273761=9, srv1954432928=47, srv1166863823=11, srv424357571=70, srv818589027=91, srv1717713807=37, srv1971278706=48, srv667751141=84, srv719411698=86, srv1504038393=28, srv1592475274=31, srv1425804394=24, srv1337364081=20, srv1297969997=18, srv1677042635=35, srv1702701761=36, srv355058911=66, srv974907539=99, srv1557090697=29, srv752622305=89, srv1351263768=23, srv323057992=63, srv913800862=94, srv434819608=71, srv103040898=4, srv1948618655=45, srv75144734=88, srv1264406980=16, srv1755914834=39, srv201333103=53, srv1972603170=49, srv711554003=85, srv1989810761=52, srv355706026=67, srv1344909140=22, srv388175903=69, srv813086408=90, srv639008775=82, srv2073761099=57, srv518668489=74, srv34769195=65, srv263531896=62, srv648604316=83, srv1000422246=0, srv1949975054=46, srv1454456=25, srv2021112311=54, srv578408332=79, srv1232954782=14, srv932573851=95, srv568909358=78, srv249651420=61, srv1476658600=26, srv567692858=77, srv969020705=98, srv1566343790=30, srv368747626=68, srv635040161=81, srv1251933708=15, srv446095969=72, srv1161895927=10, srv1325018111=19, srv1036833636=5, srv1825850676=41, srv1794868146=40, srv218671853=59, srv1205938727=13, srv1338914418=21, srv1830469763=42, srv333840381=64, srv1056037271=6, srv2138651010=58, srv725621880=87, srv472645754=73, srv1729959962=38, srv1942630762=43, srv819754298=92, srv19859932=51, srv1028844665=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-06 09:57:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-06 09:57:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-06 09:57:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-06 09:57:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-06 09:57:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-06 09:57:01,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-06 09:57:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-06 09:57:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-06 09:57:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-06 09:57:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-06 09:57:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-06 09:57:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-06 09:57:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-06 09:57:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-06 09:57:01,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-06 09:57:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-06 09:57:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-06 09:57:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-06 09:57:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-06 09:57:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-06 09:57:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-06 09:57:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-06 09:57:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-06 09:57:01,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-06 09:57:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-06 09:57:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-06 09:57:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-06 09:57:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-06 09:57:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-06 09:57:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-06 09:57:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-06 09:57:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-06 09:57:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-06 09:57:01,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-06 09:57:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-06 09:57:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-06 09:57:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-06 09:57:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-06 09:57:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-06 09:57:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-06 09:57:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-06 09:57:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-06 09:57:01,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-06 09:57:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-06 09:57:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-06 09:57:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-06 09:57:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-06 09:57:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-06 09:57:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-06 09:57:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-06 09:57:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-06 09:57:01,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-06 09:57:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-06 09:57:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-06 09:57:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-06 09:57:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-06 09:57:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-06 09:57:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-06 09:57:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-06 09:57:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-06 09:57:01,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-06 09:57:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-06 09:57:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-06 09:57:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-06 09:57:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-06 09:57:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-06 09:57:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-06 09:57:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-06 09:57:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-06 09:57:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-06 09:57:01,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-06 09:57:01,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-06 09:57:01,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-06 09:57:01,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-06 09:57:01,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-06 09:57:01,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-06 09:57:01,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-06 09:57:01,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-06 09:57:01,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-06 09:57:01,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-06 09:57:01,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-06 09:57:01,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-06 09:57:01,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-06 09:57:01,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-06 09:57:01,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-06 09:57:01,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-06 09:57:01,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-06 09:57:01,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-06 09:57:01,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-06 09:57:01,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-06 09:57:01,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-06 09:57:01,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-06 09:57:01,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-06 09:57:01,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-06 09:57:01,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-06 09:57:01,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-06 09:57:01,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-06 09:57:01,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-06 09:57:01,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-06 09:57:01,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-06 09:57:01,867 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-06 09:57:01,997 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00767691535093358); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.013121208233216896); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00767691535093358); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-06 09:57:01,997 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-06 09:57:01,997 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.161682481163929E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00767691535093358); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.013121208233216896); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00767691535093358); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-06 09:57:02,264 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 381 ms to try 5000 different iterations.  Found a solution that moves 276 regions; Going from a computed imbalance of 5.161682481163929E-4 to a new imbalance of 2.083565335676242E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.030666666666666665, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-06 09:57:02,265 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1060846971:83 , srv1454456:83 , srv1677042635:83 , srv2064267976:83 , srv523128973:83 , srv578408332:83 , srv648604316:83 , srv725621880:83 , srv955526783:83 , srv1003634171:84 , srv1028844665:84 , srv1166863823:84 , srv120580797:84 , srv1251933708:84 , srv1264406980:84 , srv1285789177:84 , srv1344909140:84 , srv1476658600:84 , srv1504038393:84 , srv1608894804:84 , srv1655433505:84 , srv1671705647:84 , srv1702701761:84 , srv1755914834:84 , srv1942630762:84 , srv1948618655:84 , srv201333103:84 , srv2021112311:84 , srv249651420:84 , srv323057992:84 , srv355058911:84 , srv434819608:84 , srv446095969:84 , srv472645754:84 , srv568909358:84 , srv711554003:84 , srv1000422246:85 , srv1019689032:85 , srv103040898:85 , srv1036833636:85 , srv1119273761:85 , srv1161895927:85 , srv1205938727:85 , srv1232954782:85 , srv1325018111:85 , srv1337364081:85 , srv1338914418:85 , srv1351263768:85 , srv1425804394:85 , srv1485374761:85 , srv1566343790:85 , srv1717713807:85 , srv1729959962:85 , srv1825850676:85 , srv1830469763:85 , srv1949975054:85 , srv1954432928:85 , srv1972603170:85 , srv1981560900:85 , srv19859932:85 , srv2041973834:85 , srv2073761099:85 , srv2138651010:85 , srv218671853:85 , srv232604783:85 , srv263531896:85 , srv333840381:85 , srv34769195:85 , srv355706026:85 , srv368747626:85 , srv388175903:85 , srv424357571:85 , srv518668489:85 , srv567692858:85 , srv604382155:85 , srv635040161:85 , srv639008775:85 , srv667751141:85 , srv813086408:85 , srv818589027:85 , srv819754298:85 , srv864665878:85 , srv932573851:85 , srv962482486:85 , srv969020705:85 , srv1056037271:86 , srv1108914261:86 , srv1297969997:86 , srv1557090697:86 , srv1794868146:86 , srv1943153373:86 , srv1971278706:86 , srv1989810761:86 , srv539887239:86 , srv719411698:86 , srv75144734:86 , srv752622305:86 , srv913800862:86 , srv974907539:86 , srv1592475274:616 }
2023-03-06 09:57:02,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-06 09:57:02,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1019689032=2, srv1981560900=50, srv539887239=76, srv1003634171=1, srv2064267976=56, srv1485374761=27, srv864665878=93, srv955526783=96, srv1943153373=44, srv2041973834=55, srv1671705647=34, srv1060846971=7, srv1108914261=8, srv232604783=60, srv523128973=75, srv1655433505=33, srv1285789177=17, srv1608894804=32, srv604382155=80, srv962482486=97, srv120580797=12, srv1119273761=9, srv1954432928=47, srv1166863823=11, srv424357571=70, srv818589027=91, srv1717713807=37, srv1971278706=48, srv667751141=84, srv719411698=86, srv1504038393=28, srv1592475274=31, srv1425804394=24, srv1337364081=20, srv1297969997=18, srv1677042635=35, srv1702701761=36, srv355058911=66, srv974907539=99, srv1557090697=29, srv752622305=89, srv1351263768=23, srv323057992=63, srv913800862=94, srv434819608=71, srv103040898=4, srv1948618655=45, srv75144734=88, srv1264406980=16, srv1755914834=39, srv201333103=53, srv1972603170=49, srv711554003=85, srv1989810761=52, srv355706026=67, srv1344909140=22, srv388175903=69, srv813086408=90, srv639008775=82, srv2073761099=57, srv518668489=74, srv34769195=65, srv263531896=62, srv648604316=83, srv1000422246=0, srv1949975054=46, srv1454456=25, srv2021112311=54, srv578408332=79, srv1232954782=14, srv932573851=95, srv568909358=78, srv249651420=61, srv1476658600=26, srv567692858=77, srv969020705=98, srv1566343790=30, srv368747626=68, srv635040161=81, srv1251933708=15, srv446095969=72, srv1161895927=10, srv1325018111=19, srv1036833636=5, srv1825850676=41, srv1794868146=40, srv218671853=59, srv1205938727=13, srv1338914418=21, srv1830469763=42, srv333840381=64, srv1056037271=6, srv2138651010=58, srv725621880=87, srv472645754=73, srv1729959962=38, srv1942630762=43, srv819754298=92, srv19859932=51, srv1028844665=3} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-06 09:57:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-06 09:57:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-06 09:57:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-06 09:57:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-06 09:57:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-06 09:57:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-06 09:57:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-06 09:57:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-06 09:57:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-06 09:57:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-06 09:57:02,285 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-06 09:57:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-06 09:57:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-06 09:57:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-06 09:57:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-06 09:57:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-06 09:57:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-06 09:57:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-06 09:57:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-06 09:57:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-06 09:57:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-06 09:57:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-06 09:57:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-06 09:57:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-06 09:57:02,286 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-06 09:57:02,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-06 09:57:02,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-06 09:57:02,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-06 09:57:02,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-06 09:57:02,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-06 09:57:02,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-06 09:57:02,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-06 09:57:02,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-06 09:57:02,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-06 09:57:02,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-06 09:57:02,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-06 09:57:02,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-06 09:57:02,287 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-06 09:57:02,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-06 09:57:02,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-06 09:57:02,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-06 09:57:02,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-06 09:57:02,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-06 09:57:02,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-06 09:57:02,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-06 09:57:02,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-06 09:57:02,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-06 09:57:02,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-06 09:57:02,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-06 09:57:02,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-06 09:57:02,288 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-06 09:57:02,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-06 09:57:02,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-06 09:57:02,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-06 09:57:02,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-06 09:57:02,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-06 09:57:02,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-06 09:57:02,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-06 09:57:02,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-06 09:57:02,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-06 09:57:02,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-06 09:57:02,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-06 09:57:02,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-06 09:57:02,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-06 09:57:02,289 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-06 09:57:02,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-06 09:57:02,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-06 09:57:02,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-06 09:57:02,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-06 09:57:02,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-06 09:57:02,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-06 09:57:02,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-06 09:57:02,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-06 09:57:02,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-06 09:57:02,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-06 09:57:02,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-06 09:57:02,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-06 09:57:02,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-06 09:57:02,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-06 09:57:02,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-06 09:57:02,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-06 09:57:02,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-06 09:57:02,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-06 09:57:02,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-06 09:57:02,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-06 09:57:02,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-06 09:57:02,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-06 09:57:02,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-06 09:57:02,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-06 09:57:02,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-06 09:57:02,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-06 09:57:02,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-06 09:57:02,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-06 09:57:02,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-06 09:57:02,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-06 09:57:02,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-06 09:57:02,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-06 09:57:02,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-06 09:57:02,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-06 09:57:02,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-06 09:57:02,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-06 09:57:02,422 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8903060050617428E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.002118805753879094); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-06 09:57:02,423 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1060846971:83 , srv1454456:83 , srv1677042635:83 , srv2064267976:83 , srv523128973:83 , srv578408332:83 , srv648604316:83 , srv725621880:83 , srv955526783:83 , srv1003634171:84 , srv1028844665:84 , srv1166863823:84 , srv120580797:84 , srv1251933708:84 , srv1264406980:84 , srv1285789177:84 , srv1344909140:84 , srv1476658600:84 , srv1504038393:84 , srv1608894804:84 , srv1655433505:84 , srv1671705647:84 , srv1702701761:84 , srv1755914834:84 , srv1942630762:84 , srv1948618655:84 , srv201333103:84 , srv2021112311:84 , srv249651420:84 , srv323057992:84 , srv355058911:84 , srv434819608:84 , srv446095969:84 , srv472645754:84 , srv568909358:84 , srv711554003:84 , srv1000422246:85 , srv1019689032:85 , srv103040898:85 , srv1036833636:85 , srv1119273761:85 , srv1161895927:85 , srv1205938727:85 , srv1232954782:85 , srv1325018111:85 , srv1337364081:85 , srv1338914418:85 , srv1351263768:85 , srv1425804394:85 , srv1485374761:85 , srv1566343790:85 , srv1717713807:85 , srv1729959962:85 , srv1825850676:85 , srv1830469763:85 , srv1949975054:85 , srv1954432928:85 , srv1972603170:85 , srv1981560900:85 , srv19859932:85 , srv2041973834:85 , srv2073761099:85 , srv2138651010:85 , srv218671853:85 , srv232604783:85 , srv263531896:85 , srv333840381:85 , srv34769195:85 , srv355706026:85 , srv368747626:85 , srv388175903:85 , srv424357571:85 , srv518668489:85 , srv567692858:85 , srv604382155:85 , srv635040161:85 , srv639008775:85 , srv667751141:85 , srv813086408:85 , srv818589027:85 , srv819754298:85 , srv864665878:85 , srv932573851:85 , srv962482486:85 , srv969020705:85 , srv1056037271:86 , srv1108914261:86 , srv1297969997:86 , srv1557090697:86 , srv1794868146:86 , srv1943153373:86 , srv1971278706:86 , srv1989810761:86 , srv539887239:86 , srv719411698:86 , srv75144734:86 , srv752622305:86 , srv913800862:86 , srv974907539:86 , srv1592475274:616 }
2023-03-06 09:57:02,454 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=710 (was 737), ProcessCount=181 (was 181), AvailableMemoryMB=6604 (was 7090)
2023-03-06 09:57:02,462 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=710, ProcessCount=181, AvailableMemoryMB=6604
2023-03-06 09:57:02,463 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-06 09:57:02,463 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-06 09:57:02,473 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1050862185:84 , srv1091574421:84 , srv1112553290:84 , srv1215559153:84 , srv128174268:84 , srv1381063078:84 , srv1420909616:84 , srv1456487967:84 , srv1465127511:84 , srv1476681650:84 , srv1528539898:84 , srv1702058936:84 , srv1758274350:84 , srv1798355864:84 , srv1934528804:84 , srv2083770097:84 , srv213307087:84 , srv281505155:84 , srv336117352:84 , srv340971048:84 , srv443069055:84 , srv49820011:84 , srv659486709:84 , srv725985519:84 , srv758708403:84 , srv79867665:84 , srv852999098:84 , srv860473557:84 , srv867540236:84 , srv1204249684:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-06 09:57:02,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-06 09:57:02,474 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1112553290=2, srv1465127511=9, srv725985519=24, srv1050862185=0, srv1091574421=1, srv2083770097=16, srv1381063078=6, srv1702058936=12, srv1758274350=13, srv1798355864=14, srv1456487967=8, srv1215559153=4, srv852999098=27, srv758708403=25, srv860473557=28, srv1476681650=10, srv1204249684=3, srv443069055=21, srv128174268=5, srv213307087=17, srv659486709=23, srv336117352=19, srv1528539898=11, srv1420909616=7, srv340971048=20, srv49820011=22, srv79867665=26, srv281505155=18, srv867540236=29, srv1934528804=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-06 09:57:02,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-06 09:57:02,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-06 09:57:02,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-06 09:57:02,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-06 09:57:02,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-06 09:57:02,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-06 09:57:02,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-06 09:57:02,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-06 09:57:02,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-06 09:57:02,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-06 09:57:02,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-06 09:57:02,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-06 09:57:02,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-06 09:57:02,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-06 09:57:02,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-06 09:57:02,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-06 09:57:02,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-06 09:57:02,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-06 09:57:02,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-06 09:57:02,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-06 09:57:02,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-06 09:57:02,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-06 09:57:02,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-06 09:57:02,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-06 09:57:02,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-06 09:57:02,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-06 09:57:02,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-06 09:57:02,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-06 09:57:02,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-06 09:57:02,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-06 09:57:02,488 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-06 09:57:02,502 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0973644495883743, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-06 09:57:02,503 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912499835465691, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0973644495883743, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-06 09:59:02,491 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 8298944 different iterations.  Found a solution that moves 1840 regions; Going from a computed imbalance of 0.9912499835465691 to a new imbalance of 4.2946517914333035E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6814814814814815, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-06 09:59:02,498 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1050862185:90 , srv1091574421:90 , srv1112553290:90 , srv1204249684:90 , srv1215559153:90 , srv128174268:90 , srv1381063078:90 , srv1420909616:90 , srv1456487967:90 , srv1465127511:90 , srv1476681650:90 , srv1528539898:90 , srv1702058936:90 , srv1758274350:90 , srv1798355864:90 , srv1934528804:90 , srv2083770097:90 , srv213307087:90 , srv281505155:90 , srv336117352:90 , srv340971048:90 , srv443069055:90 , srv49820011:90 , srv659486709:90 , srv725985519:90 , srv758708403:90 , srv79867665:90 , srv852999098:90 , srv860473557:90 , srv867540236:90 }
2023-03-06 09:59:02,499 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-06 09:59:02,500 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1112553290=2, srv1465127511=9, srv725985519=24, srv1050862185=0, srv1091574421=1, srv2083770097=16, srv1381063078=6, srv1702058936=12, srv1758274350=13, srv1798355864=14, srv1456487967=8, srv1215559153=4, srv852999098=27, srv758708403=25, srv860473557=28, srv1476681650=10, srv1204249684=3, srv443069055=21, srv128174268=5, srv213307087=17, srv659486709=23, srv336117352=19, srv1528539898=11, srv1420909616=7, srv340971048=20, srv49820011=22, srv79867665=26, srv281505155=18, srv867540236=29, srv1934528804=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-06 09:59:02,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-06 09:59:02,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-06 09:59:02,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-06 09:59:02,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-06 09:59:02,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-06 09:59:02,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-06 09:59:02,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-06 09:59:02,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-06 09:59:02,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-06 09:59:02,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-06 09:59:02,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-06 09:59:02,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-06 09:59:02,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-06 09:59:02,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-06 09:59:02,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-06 09:59:02,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-06 09:59:02,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-06 09:59:02,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-06 09:59:02,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-06 09:59:02,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-06 09:59:02,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-06 09:59:02,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-06 09:59:02,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-06 09:59:02,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-06 09:59:02,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-06 09:59:02,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-06 09:59:02,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-06 09:59:02,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-06 09:59:02,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-06 09:59:02,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-06 09:59:02,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-06 09:59:02,523 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-06 09:59:02,523 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1050862185:90 , srv1091574421:90 , srv1112553290:90 , srv1204249684:90 , srv1215559153:90 , srv128174268:90 , srv1381063078:90 , srv1420909616:90 , srv1456487967:90 , srv1465127511:90 , srv1476681650:90 , srv1528539898:90 , srv1702058936:90 , srv1758274350:90 , srv1798355864:90 , srv1934528804:90 , srv2083770097:90 , srv213307087:90 , srv281505155:90 , srv336117352:90 , srv340971048:90 , srv443069055:90 , srv49820011:90 , srv659486709:90 , srv725985519:90 , srv758708403:90 , srv79867665:90 , srv852999098:90 , srv860473557:90 , srv867540236:90 }
2023-03-06 09:59:02,539 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=536 (was 710), ProcessCount=181 (was 181), AvailableMemoryMB=7018 (was 6604) - AvailableMemoryMB LEAK? -