2024-04-14 09:53:17,558 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-04-14 09:53:17,920 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-04-14 09:53:18,033 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-04-14 09:53:18,098 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-04-14 09:53:18,106 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-04-14 09:53:18,121 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=304, ProcessCount=169, AvailableMemoryMB=8613
2024-04-14 09:53:18,123 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-04-14 09:53:18,123 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-04-14 09:53:18,265 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1026917440:84 , srv1033160053:84 , srv1053400175:84 , srv1056669629:84 , srv1062928719:84 , srv1093822065:84 , srv1107863976:84 , srv1119370761:84 , srv1125962149:84 , srv1135503083:84 , srv1150054220:84 , srv1160714495:84 , srv1179794874:84 , srv118221892:84 , srv1222812910:84 , srv1246840739:84 , srv1306893056:84 , srv1310673558:84 , srv1336281591:84 , srv1351005719:84 , srv1388645040:84 , srv1390043281:84 , srv149595270:84 , srv1533958490:84 , srv1536524937:84 , srv1538072058:84 , srv1540424751:84 , srv1553364882:84 , srv1558585296:84 , srv1600661103:84 , srv1607768741:84 , srv163445875:84 , srv164587798:84 , srv165532480:84 , srv1661630029:84 , srv1675340821:84 , srv168455060:84 , srv169609168:84 , srv1709736655:84 , srv1723951352:84 , srv1773121043:84 , srv178289636:84 , srv181295235:84 , srv1822008101:84 , srv1830723803:84 , srv1857519778:84 , srv1869120651:84 , srv1872641758:84 , srv1886248326:84 , srv1888499048:84 , srv1891228156:84 , srv1962579245:84 , srv1972851627:84 , srv1983461671:84 , srv2018157570:84 , srv2030317289:84 , srv2039614665:84 , srv2053784463:84 , srv2084841018:84 , srv2115564964:84 , srv2127877831:84 , srv256702140:84 , srv266879293:84 , srv287250097:84 , srv31027378:84 , srv32716129:84 , srv337436043:84 , srv34452873:84 , srv354156757:84 , srv369887682:84 , srv375265666:84 , srv376329082:84 , srv3953202:84 , srv396654860:84 , srv410158971:84 , srv417308407:84 , srv476463117:84 , srv483322064:84 , srv534216578:84 , srv560519836:84 , srv564720889:84 , srv569556584:84 , srv583677301:84 , srv611948404:84 , srv618563107:84 , srv653065429:84 , srv654437225:84 , srv660800570:84 , srv677955032:84 , srv68686373:84 , srv750594714:84 , srv763744911:84 , srv78248640:84 , srv793001885:84 , srv794040970:84 , srv816566455:84 , srv825754633:84 , srv863863394:84 , srv987188367:84 , srv2004989792:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-04-14 09:53:18,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-14 09:53:18,283 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv164587798=32, srv1336281591=18, srv1869120651=46, srv1675340821=35, srv396654860=74, srv660800570=88, srv653065429=86, srv169609168=37, srv2018157570=55, srv1536524937=24, srv417308407=76, srv178289636=41, srv1886248326=48, srv369887682=70, srv376329082=72, srv1661630029=34, srv816566455=96, srv2053784463=58, srv560519836=80, srv1857519778=45, srv1026917440=0, srv654437225=87, srv337436043=67, srv2004989792=54, srv1538072058=25, srv534216578=79, srv1972851627=52, srv1107863976=6, srv31027378=65, srv1888499048=49, srv1607768741=30, srv1600661103=29, srv32716129=66, srv1150054220=10, srv375265666=71, srv1135503083=9, srv287250097=64, srv794040970=95, srv1033160053=1, srv78248640=93, srv1351005719=19, srv483322064=78, srv677955032=89, srv1822008101=43, srv1830723803=44, srv564720889=81, srv34452873=68, srv476463117=77, srv1310673558=17, srv1558585296=28, srv1773121043=40, srv1533958490=23, srv1125962149=8, srv68686373=90, srv1723951352=39, srv1709736655=38, srv763744911=92, srv2127877831=61, srv611948404=84, srv1246840739=15, srv825754633=97, srv181295235=42, srv1962579245=51, srv1388645040=20, srv2115564964=60, srv1540424751=26, srv1222812910=14, srv750594714=91, srv1056669629=3, srv1053400175=2, srv1306893056=16, srv1179794874=12, srv3953202=73, srv1983461671=53, srv987188367=99, srv168455060=36, srv1062928719=4, srv2039614665=57, srv793001885=94, srv1093822065=5, srv266879293=63, srv1872641758=47, srv1891228156=50, srv569556584=82, srv1160714495=11, srv863863394=98, srv2084841018=59, srv2030317289=56, srv354156757=69, srv583677301=83, srv165532480=33, srv1553364882=27, srv618563107=85, srv1119370761=7, srv256702140=62, srv1390043281=21, srv410158971=75, srv118221892=13, srv149595270=22, srv163445875=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-14 09:53:18,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-14 09:53:18,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-14 09:53:18,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-14 09:53:18,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-14 09:53:18,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-14 09:53:18,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-14 09:53:18,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-14 09:53:18,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-14 09:53:18,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-14 09:53:18,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-14 09:53:18,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-14 09:53:18,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-14 09:53:18,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-14 09:53:18,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-14 09:53:18,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-14 09:53:18,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-14 09:53:18,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-14 09:53:18,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-14 09:53:18,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-14 09:53:18,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-14 09:53:18,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-14 09:53:18,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-14 09:53:18,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-14 09:53:18,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-14 09:53:18,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-14 09:53:18,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-14 09:53:18,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-14 09:53:18,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-14 09:53:18,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-14 09:53:18,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-14 09:53:18,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-14 09:53:18,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-14 09:53:18,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-14 09:53:18,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-14 09:53:18,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-14 09:53:18,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-14 09:53:18,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-14 09:53:18,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-14 09:53:18,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-14 09:53:18,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-14 09:53:18,387 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-14 09:53:18,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-14 09:53:18,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-14 09:53:18,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-14 09:53:18,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-14 09:53:18,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-14 09:53:18,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-14 09:53:18,388 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-14 09:53:18,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-14 09:53:18,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-14 09:53:18,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-14 09:53:18,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-14 09:53:18,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-14 09:53:18,389 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-14 09:53:18,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-14 09:53:18,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-14 09:53:18,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-14 09:53:18,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-14 09:53:18,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-14 09:53:18,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-14 09:53:18,390 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-14 09:53:18,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-14 09:53:18,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-14 09:53:18,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-14 09:53:18,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-14 09:53:18,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-14 09:53:18,391 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-14 09:53:18,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-14 09:53:18,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-14 09:53:18,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-14 09:53:18,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-14 09:53:18,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-14 09:53:18,392 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-14 09:53:18,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-14 09:53:18,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-14 09:53:18,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-14 09:53:18,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-14 09:53:18,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-14 09:53:18,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-14 09:53:18,393 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-14 09:53:18,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-14 09:53:18,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-14 09:53:18,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-14 09:53:18,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-14 09:53:18,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-14 09:53:18,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-14 09:53:18,394 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-14 09:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-14 09:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-14 09:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-14 09:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-14 09:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-14 09:53:18,395 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-14 09:53:18,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-14 09:53:18,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-14 09:53:18,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-14 09:53:18,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-14 09:53:18,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-14 09:53:18,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-14 09:53:18,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-14 09:53:18,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-14 09:53:18,592 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-14 09:53:18,592 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-14 09:53:18,592 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-14 09:53:18,961 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 532 ms to try 5000 different iterations.  Found a solution that moves 4712 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1785093073815988. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039976923134973356, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05311825318615834, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5235555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039976923134973356, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1735, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24266666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-14 09:53:18,981 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv168455060:80 , srv2018157570:81 , srv618563107:81 , srv1135503083:82 , srv1558585296:82 , srv164587798:82 , srv34452873:82 , srv564720889:82 , srv611948404:82 , srv660800570:82 , srv750594714:82 , srv1388645040:83 , srv1538072058:83 , srv287250097:83 , srv534216578:83 , srv560519836:83 , srv987188367:83 , srv1062928719:84 , srv1119370761:84 , srv1536524937:84 , srv1553364882:84 , srv165532480:84 , srv1675340821:84 , srv178289636:84 , srv1830723803:84 , srv375265666:84 , srv654437225:84 , srv794040970:84 , srv825754633:84 , srv1026917440:85 , srv1888499048:85 , srv1962579245:85 , srv2053784463:85 , srv410158971:85 , srv476463117:85 , srv1033160053:86 , srv1093822065:86 , srv1107863976:86 , srv1125962149:86 , srv1306893056:86 , srv1351005719:86 , srv149595270:86 , srv1600661103:86 , srv169609168:86 , srv181295235:86 , srv1886248326:86 , srv2030317289:86 , srv2039614665:86 , srv266879293:86 , srv3953202:86 , srv583677301:86 , srv653065429:86 , srv68686373:86 , srv863863394:86 , srv118221892:87 , srv1246840739:87 , srv163445875:87 , srv1723951352:87 , srv2084841018:87 , srv31027378:87 , srv396654860:87 , srv763744911:87 , srv793001885:87 , srv1056669629:88 , srv1160714495:88 , srv1222812910:88 , srv1661630029:88 , srv1822008101:88 , srv1872641758:88 , srv354156757:88 , srv369887682:88 , srv677955032:88 , srv78248640:88 , srv816566455:88 , srv1053400175:89 , srv1179794874:89 , srv1390043281:89 , srv1709736655:89 , srv1869120651:89 , srv2127877831:89 , srv256702140:89 , srv337436043:89 , srv417308407:89 , srv483322064:89 , srv1310673558:90 , srv1533958490:90 , srv1607768741:90 , srv1773121043:90 , srv1857519778:90 , srv1983461671:90 , srv2115564964:90 , srv376329082:90 , srv569556584:91 , srv1540424751:92 , srv1891228156:92 , srv1972851627:92 , srv32716129:92 , srv1336281591:93 , srv1150054220:94 , srv2004989792:445 }
2024-04-14 09:53:18,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-14 09:53:18,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv164587798=32, srv1336281591=18, srv1869120651=46, srv1675340821=35, srv396654860=74, srv660800570=88, srv653065429=86, srv169609168=37, srv2018157570=55, srv1536524937=24, srv417308407=76, srv178289636=41, srv1886248326=48, srv369887682=70, srv376329082=72, srv1661630029=34, srv816566455=96, srv2053784463=58, srv560519836=80, srv1857519778=45, srv1026917440=0, srv654437225=87, srv337436043=67, srv2004989792=54, srv1538072058=25, srv534216578=79, srv1972851627=52, srv1107863976=6, srv31027378=65, srv1888499048=49, srv1607768741=30, srv1600661103=29, srv32716129=66, srv1150054220=10, srv375265666=71, srv1135503083=9, srv287250097=64, srv794040970=95, srv1033160053=1, srv78248640=93, srv1351005719=19, srv483322064=78, srv677955032=89, srv1822008101=43, srv1830723803=44, srv564720889=81, srv34452873=68, srv476463117=77, srv1310673558=17, srv1558585296=28, srv1773121043=40, srv1533958490=23, srv1125962149=8, srv68686373=90, srv1723951352=39, srv1709736655=38, srv763744911=92, srv2127877831=61, srv611948404=84, srv1246840739=15, srv825754633=97, srv181295235=42, srv1962579245=51, srv1388645040=20, srv2115564964=60, srv1540424751=26, srv1222812910=14, srv750594714=91, srv1056669629=3, srv1053400175=2, srv1306893056=16, srv1179794874=12, srv3953202=73, srv1983461671=53, srv987188367=99, srv168455060=36, srv1062928719=4, srv2039614665=57, srv793001885=94, srv1093822065=5, srv266879293=63, srv1872641758=47, srv1891228156=50, srv569556584=82, srv1160714495=11, srv863863394=98, srv2084841018=59, srv2030317289=56, srv354156757=69, srv583677301=83, srv165532480=33, srv1553364882=27, srv618563107=85, srv1119370761=7, srv256702140=62, srv1390043281=21, srv410158971=75, srv118221892=13, srv149595270=22, srv163445875=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-14 09:53:18,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-14 09:53:18,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-14 09:53:18,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-14 09:53:18,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-14 09:53:18,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-14 09:53:18,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-14 09:53:18,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-14 09:53:18,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-14 09:53:18,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-14 09:53:18,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-14 09:53:18,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-14 09:53:18,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-14 09:53:18,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-14 09:53:18,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-14 09:53:18,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-14 09:53:18,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-14 09:53:18,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-14 09:53:18,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-14 09:53:18,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-14 09:53:18,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-14 09:53:19,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-14 09:53:19,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-14 09:53:19,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-14 09:53:19,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-14 09:53:19,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-14 09:53:19,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-14 09:53:19,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-14 09:53:19,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-14 09:53:19,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-14 09:53:19,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-14 09:53:19,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-14 09:53:19,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-14 09:53:19,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-14 09:53:19,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-14 09:53:19,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-14 09:53:19,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-14 09:53:19,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-14 09:53:19,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-14 09:53:19,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-14 09:53:19,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-14 09:53:19,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-14 09:53:19,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-14 09:53:19,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-14 09:53:19,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-14 09:53:19,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-14 09:53:19,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-14 09:53:19,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-14 09:53:19,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-14 09:53:19,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-14 09:53:19,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-14 09:53:19,002 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-14 09:53:19,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-14 09:53:19,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-14 09:53:19,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-14 09:53:19,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-14 09:53:19,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-14 09:53:19,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-14 09:53:19,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-14 09:53:19,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-14 09:53:19,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-14 09:53:19,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-14 09:53:19,003 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-14 09:53:19,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-14 09:53:19,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-14 09:53:19,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-14 09:53:19,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-14 09:53:19,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-14 09:53:19,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-14 09:53:19,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-14 09:53:19,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-14 09:53:19,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-14 09:53:19,004 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-14 09:53:19,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-14 09:53:19,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-14 09:53:19,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-14 09:53:19,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-14 09:53:19,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-14 09:53:19,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-14 09:53:19,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-14 09:53:19,005 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-14 09:53:19,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-14 09:53:19,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-14 09:53:19,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-14 09:53:19,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-14 09:53:19,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-14 09:53:19,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-14 09:53:19,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-14 09:53:19,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-14 09:53:19,006 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-14 09:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-14 09:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-14 09:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-14 09:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-14 09:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-14 09:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-14 09:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-14 09:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-14 09:53:19,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-14 09:53:19,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-14 09:53:19,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-14 09:53:19,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-14 09:53:19,175 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039976923134973356, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05311825318615834, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039976923134973356, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1735, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24266666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-14 09:53:19,175 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-14 09:53:19,175 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17847631325240113, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039976923134973356, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05311825318615834, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039976923134973356, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1735, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24266666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-14 09:53:19,517 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 472 ms to try 5000 different iterations.  Found a solution that moves 2709 regions; Going from a computed imbalance of 0.17847631325240113 to a new imbalance of 0.015589546820079419. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020816360463599116); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03179267857187265, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.301, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020816360463599116); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04608333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-14 09:53:19,535 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1150054220:77 , srv1972851627:79 , srv1891228156:80 , srv32716129:80 , srv1310673558:81 , srv1857519778:81 , srv569556584:81 , srv1540424751:82 , srv1709736655:82 , srv1822008101:82 , srv1869120651:82 , srv2115564964:82 , srv1336281591:83 , srv1390043281:83 , srv1533958490:83 , srv1607768741:83 , srv1773121043:83 , srv1983461671:83 , srv2127877831:83 , srv256702140:83 , srv376329082:83 , srv417308407:83 , srv78248640:83 , srv1033160053:84 , srv1053400175:84 , srv1107863976:84 , srv1160714495:84 , srv1179794874:84 , srv118221892:84 , srv1351005719:84 , srv1661630029:84 , srv2030317289:84 , srv2084841018:84 , srv354156757:84 , srv583677301:84 , srv68686373:84 , srv793001885:84 , srv1056669629:85 , srv1222812910:85 , srv149595270:85 , srv169609168:85 , srv1872641758:85 , srv2039614665:85 , srv266879293:85 , srv31027378:85 , srv337436043:85 , srv369887682:85 , srv483322064:85 , srv677955032:85 , srv816566455:85 , srv1246840739:86 , srv1306893056:86 , srv1600661103:86 , srv287250097:86 , srv3953202:86 , srv396654860:86 , srv476463117:86 , srv763744911:86 , srv825754633:86 , srv863863394:86 , srv1125962149:87 , srv1135503083:87 , srv1538072058:87 , srv163445875:87 , srv1675340821:87 , srv2053784463:87 , srv410158971:87 , srv653065429:87 , srv1026917440:88 , srv1093822065:88 , srv1119370761:88 , srv165532480:88 , srv1723951352:88 , srv178289636:88 , srv181295235:88 , srv1830723803:88 , srv1886248326:88 , srv1888499048:88 , srv375265666:88 , srv560519836:88 , srv564720889:88 , srv660800570:88 , srv1062928719:89 , srv1536524937:89 , srv164587798:89 , srv1962579245:89 , srv654437225:89 , srv794040970:89 , srv1388645040:90 , srv1553364882:90 , srv611948404:90 , srv987188367:90 , srv1558585296:91 , srv2018157570:91 , srv534216578:91 , srv750594714:91 , srv34452873:92 , srv618563107:92 , srv168455060:93 , srv2004989792:514 }
2024-04-14 09:53:19,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-14 09:53:19,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv164587798=32, srv1336281591=18, srv1869120651=46, srv1675340821=35, srv396654860=74, srv660800570=88, srv653065429=86, srv169609168=37, srv2018157570=55, srv1536524937=24, srv417308407=76, srv178289636=41, srv1886248326=48, srv369887682=70, srv376329082=72, srv1661630029=34, srv816566455=96, srv2053784463=58, srv560519836=80, srv1857519778=45, srv1026917440=0, srv654437225=87, srv337436043=67, srv2004989792=54, srv1538072058=25, srv534216578=79, srv1972851627=52, srv1107863976=6, srv31027378=65, srv1888499048=49, srv1607768741=30, srv1600661103=29, srv32716129=66, srv1150054220=10, srv375265666=71, srv1135503083=9, srv287250097=64, srv794040970=95, srv1033160053=1, srv78248640=93, srv1351005719=19, srv483322064=78, srv677955032=89, srv1822008101=43, srv1830723803=44, srv564720889=81, srv34452873=68, srv476463117=77, srv1310673558=17, srv1558585296=28, srv1773121043=40, srv1533958490=23, srv1125962149=8, srv68686373=90, srv1723951352=39, srv1709736655=38, srv763744911=92, srv2127877831=61, srv611948404=84, srv1246840739=15, srv825754633=97, srv181295235=42, srv1962579245=51, srv1388645040=20, srv2115564964=60, srv1540424751=26, srv1222812910=14, srv750594714=91, srv1056669629=3, srv1053400175=2, srv1306893056=16, srv1179794874=12, srv3953202=73, srv1983461671=53, srv987188367=99, srv168455060=36, srv1062928719=4, srv2039614665=57, srv793001885=94, srv1093822065=5, srv266879293=63, srv1872641758=47, srv1891228156=50, srv569556584=82, srv1160714495=11, srv863863394=98, srv2084841018=59, srv2030317289=56, srv354156757=69, srv583677301=83, srv165532480=33, srv1553364882=27, srv618563107=85, srv1119370761=7, srv256702140=62, srv1390043281=21, srv410158971=75, srv118221892=13, srv149595270=22, srv163445875=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-14 09:53:19,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-14 09:53:19,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-14 09:53:19,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-14 09:53:19,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-14 09:53:19,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-14 09:53:19,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-14 09:53:19,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-14 09:53:19,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-14 09:53:19,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-14 09:53:19,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-14 09:53:19,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-14 09:53:19,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-14 09:53:19,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-14 09:53:19,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-14 09:53:19,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-14 09:53:19,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-14 09:53:19,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-14 09:53:19,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-14 09:53:19,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-14 09:53:19,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-14 09:53:19,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-14 09:53:19,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-14 09:53:19,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-14 09:53:19,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-14 09:53:19,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-14 09:53:19,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-14 09:53:19,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-14 09:53:19,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-14 09:53:19,567 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-14 09:53:19,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-14 09:53:19,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-14 09:53:19,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-14 09:53:19,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-14 09:53:19,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-14 09:53:19,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-14 09:53:19,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-14 09:53:19,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-14 09:53:19,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-14 09:53:19,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-14 09:53:19,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-14 09:53:19,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-14 09:53:19,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-14 09:53:19,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-14 09:53:19,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-14 09:53:19,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-14 09:53:19,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-14 09:53:19,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-14 09:53:19,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-14 09:53:19,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-14 09:53:19,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-14 09:53:19,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-14 09:53:19,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-14 09:53:19,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-14 09:53:19,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-14 09:53:19,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-14 09:53:19,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-14 09:53:19,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-14 09:53:19,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-14 09:53:19,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-14 09:53:19,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-14 09:53:19,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-14 09:53:19,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-14 09:53:19,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-14 09:53:19,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-14 09:53:19,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-14 09:53:19,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-14 09:53:19,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-14 09:53:19,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-14 09:53:19,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-14 09:53:19,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-14 09:53:19,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-14 09:53:19,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-14 09:53:19,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-14 09:53:19,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-14 09:53:19,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-14 09:53:19,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-14 09:53:19,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-14 09:53:19,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-14 09:53:19,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-14 09:53:19,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-14 09:53:19,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-14 09:53:19,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-14 09:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-14 09:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-14 09:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-14 09:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-14 09:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-14 09:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-14 09:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-14 09:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-14 09:53:19,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-14 09:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-14 09:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-14 09:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-14 09:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-14 09:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-14 09:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-14 09:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-14 09:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-14 09:53:19,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-14 09:53:19,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-14 09:53:19,711 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020816360463599116); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03179267857187265, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020816360463599116); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04608333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-14 09:53:19,712 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-14 09:53:19,712 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015570577996650627, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020816360463599116); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03179267857187265, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020816360463599116); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.012416666666666666); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04608333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-14 09:53:19,965 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 369 ms to try 5000 different iterations.  Found a solution that moves 1017 regions; Going from a computed imbalance of 0.015570577996650627 to a new imbalance of 7.366100568181896E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006106262743597928); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00552516942673832); MoveCostFunction : (multiplier=7.0, imbalance=0.113, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006106262743597928); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0033333333333333335); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-14 09:53:19,968 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1336281591:82 , srv1093822065:83 , srv1540424751:83 , srv163445875:83 , srv1723951352:83 , srv181295235:83 , srv1886248326:83 , srv337436043:83 , srv483322064:83 , srv1026917440:84 , srv1053400175:84 , srv1062928719:84 , srv1125962149:84 , srv1222812910:84 , srv1388645040:84 , srv1553364882:84 , srv1607768741:84 , srv168455060:84 , srv1773121043:84 , srv178289636:84 , srv1872641758:84 , srv1962579245:84 , srv34452873:84 , srv369887682:84 , srv376329082:84 , srv534216578:84 , srv677955032:84 , srv794040970:84 , srv1056669629:85 , srv1160714495:85 , srv1179794874:85 , srv1306893056:85 , srv1536524937:85 , srv1558585296:85 , srv1600661103:85 , srv164587798:85 , srv1661630029:85 , srv1888499048:85 , srv1891228156:85 , srv1983461671:85 , srv2018157570:85 , srv2084841018:85 , srv2115564964:85 , srv266879293:85 , srv32716129:85 , srv396654860:85 , srv410158971:85 , srv569556584:85 , srv611948404:85 , srv618563107:85 , srv653065429:85 , srv654437225:85 , srv750594714:85 , srv763744911:85 , srv816566455:85 , srv863863394:85 , srv987188367:85 , srv1119370761:86 , srv1150054220:86 , srv118221892:86 , srv1246840739:86 , srv1310673558:86 , srv149595270:86 , srv1533958490:86 , srv1675340821:86 , srv1709736655:86 , srv1830723803:86 , srv1857519778:86 , srv1869120651:86 , srv2039614665:86 , srv2053784463:86 , srv2127877831:86 , srv256702140:86 , srv31027378:86 , srv354156757:86 , srv375265666:86 , srv3953202:86 , srv417308407:86 , srv560519836:86 , srv1033160053:87 , srv1107863976:87 , srv1351005719:87 , srv165532480:87 , srv169609168:87 , srv1972851627:87 , srv2030317289:87 , srv476463117:87 , srv564720889:87 , srv583677301:87 , srv68686373:87 , srv78248640:87 , srv793001885:87 , srv1135503083:88 , srv1390043281:88 , srv1538072058:88 , srv1822008101:88 , srv660800570:88 , srv825754633:88 , srv287250097:89 , srv2004989792:553 }
2024-04-14 09:53:19,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-14 09:53:19,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv164587798=32, srv1336281591=18, srv1869120651=46, srv1675340821=35, srv396654860=74, srv660800570=88, srv653065429=86, srv169609168=37, srv2018157570=55, srv1536524937=24, srv417308407=76, srv178289636=41, srv1886248326=48, srv369887682=70, srv376329082=72, srv1661630029=34, srv816566455=96, srv2053784463=58, srv560519836=80, srv1857519778=45, srv1026917440=0, srv654437225=87, srv337436043=67, srv2004989792=54, srv1538072058=25, srv534216578=79, srv1972851627=52, srv1107863976=6, srv31027378=65, srv1888499048=49, srv1607768741=30, srv1600661103=29, srv32716129=66, srv1150054220=10, srv375265666=71, srv1135503083=9, srv287250097=64, srv794040970=95, srv1033160053=1, srv78248640=93, srv1351005719=19, srv483322064=78, srv677955032=89, srv1822008101=43, srv1830723803=44, srv564720889=81, srv34452873=68, srv476463117=77, srv1310673558=17, srv1558585296=28, srv1773121043=40, srv1533958490=23, srv1125962149=8, srv68686373=90, srv1723951352=39, srv1709736655=38, srv763744911=92, srv2127877831=61, srv611948404=84, srv1246840739=15, srv825754633=97, srv181295235=42, srv1962579245=51, srv1388645040=20, srv2115564964=60, srv1540424751=26, srv1222812910=14, srv750594714=91, srv1056669629=3, srv1053400175=2, srv1306893056=16, srv1179794874=12, srv3953202=73, srv1983461671=53, srv987188367=99, srv168455060=36, srv1062928719=4, srv2039614665=57, srv793001885=94, srv1093822065=5, srv266879293=63, srv1872641758=47, srv1891228156=50, srv569556584=82, srv1160714495=11, srv863863394=98, srv2084841018=59, srv2030317289=56, srv354156757=69, srv583677301=83, srv165532480=33, srv1553364882=27, srv618563107=85, srv1119370761=7, srv256702140=62, srv1390043281=21, srv410158971=75, srv118221892=13, srv149595270=22, srv163445875=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-14 09:53:19,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-14 09:53:19,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-14 09:53:19,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-14 09:53:19,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-14 09:53:19,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-14 09:53:19,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-14 09:53:19,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-14 09:53:19,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-14 09:53:19,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-14 09:53:19,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-14 09:53:19,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-14 09:53:19,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-14 09:53:19,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-14 09:53:19,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-14 09:53:19,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-14 09:53:19,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-14 09:53:19,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-14 09:53:19,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-14 09:53:19,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-14 09:53:19,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-14 09:53:19,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-14 09:53:19,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-14 09:53:19,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-14 09:53:19,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-14 09:53:19,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-14 09:53:19,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-14 09:53:19,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-14 09:53:19,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-14 09:53:19,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-14 09:53:19,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-14 09:53:19,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-14 09:53:19,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-14 09:53:19,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-14 09:53:19,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-14 09:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-14 09:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-14 09:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-14 09:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-14 09:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-14 09:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-14 09:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-14 09:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-14 09:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-14 09:53:19,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-14 09:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-14 09:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-14 09:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-14 09:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-14 09:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-14 09:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-14 09:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-14 09:53:19,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-14 09:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-14 09:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-14 09:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-14 09:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-14 09:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-14 09:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-14 09:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-14 09:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-14 09:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-14 09:53:19,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-14 09:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-14 09:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-14 09:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-14 09:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-14 09:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-14 09:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-14 09:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-14 09:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-14 09:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-14 09:53:19,997 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-14 09:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-14 09:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-14 09:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-14 09:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-14 09:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-14 09:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-14 09:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-14 09:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-14 09:53:19,998 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-14 09:53:19,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-14 09:53:19,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-14 09:53:19,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-14 09:53:19,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-14 09:53:19,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-14 09:53:19,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-14 09:53:19,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-14 09:53:19,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-14 09:53:19,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-14 09:53:19,999 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-14 09:53:20,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-14 09:53:20,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-14 09:53:20,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-14 09:53:20,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-14 09:53:20,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-14 09:53:20,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-14 09:53:20,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-14 09:53:20,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-14 09:53:20,000 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-14 09:53:20,001 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-14 09:53:20,146 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006106262743597928); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00552516942673832); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006106262743597928); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0033333333333333335); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-14 09:53:20,146 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-04-14 09:53:20,146 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.29488870614025E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006106262743597928); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00552516942673832); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006106262743597928); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0033333333333333335); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-04-14 09:53:20,537 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 524 ms to try 5000 different iterations.  Found a solution that moves 243 regions; Going from a computed imbalance of 7.29488870614025E-4 to a new imbalance of 1.904671861351497E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.027, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-14 09:53:20,538 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1246840739:83 , srv1390043281:83 , srv1533958490:83 , srv1056669629:84 , srv1119370761:84 , srv1179794874:84 , srv1336281591:84 , srv1536524937:84 , srv1538072058:84 , srv1553364882:84 , srv1558585296:84 , srv1607768741:84 , srv165532480:84 , srv1661630029:84 , srv169609168:84 , srv1822008101:84 , srv1830723803:84 , srv1888499048:84 , srv1962579245:84 , srv1972851627:84 , srv1983461671:84 , srv2053784463:84 , srv2127877831:84 , srv256702140:84 , srv287250097:84 , srv31027378:84 , srv34452873:84 , srv375265666:84 , srv3953202:84 , srv396654860:84 , srv417308407:84 , srv476463117:84 , srv483322064:84 , srv534216578:84 , srv560519836:84 , srv618563107:84 , srv653065429:84 , srv654437225:84 , srv660800570:84 , srv750594714:84 , srv763744911:84 , srv78248640:84 , srv793001885:84 , srv816566455:84 , srv825754633:84 , srv863863394:84 , srv987188367:84 , srv1026917440:85 , srv1033160053:85 , srv1053400175:85 , srv1062928719:85 , srv1093822065:85 , srv1107863976:85 , srv1125962149:85 , srv1135503083:85 , srv1150054220:85 , srv1160714495:85 , srv118221892:85 , srv1222812910:85 , srv1310673558:85 , srv1351005719:85 , srv1388645040:85 , srv149595270:85 , srv1540424751:85 , srv1600661103:85 , srv1675340821:85 , srv168455060:85 , srv1709736655:85 , srv1723951352:85 , srv1773121043:85 , srv178289636:85 , srv181295235:85 , srv1857519778:85 , srv1869120651:85 , srv1872641758:85 , srv1886248326:85 , srv1891228156:85 , srv2018157570:85 , srv2030317289:85 , srv2039614665:85 , srv2115564964:85 , srv32716129:85 , srv337436043:85 , srv354156757:85 , srv369887682:85 , srv376329082:85 , srv410158971:85 , srv564720889:85 , srv569556584:85 , srv583677301:85 , srv611948404:85 , srv677955032:85 , srv68686373:85 , srv794040970:85 , srv1306893056:86 , srv163445875:86 , srv164587798:86 , srv2084841018:86 , srv266879293:86 , srv2004989792:630 }
2024-04-14 09:53:20,542 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-14 09:53:20,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv164587798=32, srv1336281591=18, srv1869120651=46, srv1675340821=35, srv396654860=74, srv660800570=88, srv653065429=86, srv169609168=37, srv2018157570=55, srv1536524937=24, srv417308407=76, srv178289636=41, srv1886248326=48, srv369887682=70, srv376329082=72, srv1661630029=34, srv816566455=96, srv2053784463=58, srv560519836=80, srv1857519778=45, srv1026917440=0, srv654437225=87, srv337436043=67, srv2004989792=54, srv1538072058=25, srv534216578=79, srv1972851627=52, srv1107863976=6, srv31027378=65, srv1888499048=49, srv1607768741=30, srv1600661103=29, srv32716129=66, srv1150054220=10, srv375265666=71, srv1135503083=9, srv287250097=64, srv794040970=95, srv1033160053=1, srv78248640=93, srv1351005719=19, srv483322064=78, srv677955032=89, srv1822008101=43, srv1830723803=44, srv564720889=81, srv34452873=68, srv476463117=77, srv1310673558=17, srv1558585296=28, srv1773121043=40, srv1533958490=23, srv1125962149=8, srv68686373=90, srv1723951352=39, srv1709736655=38, srv763744911=92, srv2127877831=61, srv611948404=84, srv1246840739=15, srv825754633=97, srv181295235=42, srv1962579245=51, srv1388645040=20, srv2115564964=60, srv1540424751=26, srv1222812910=14, srv750594714=91, srv1056669629=3, srv1053400175=2, srv1306893056=16, srv1179794874=12, srv3953202=73, srv1983461671=53, srv987188367=99, srv168455060=36, srv1062928719=4, srv2039614665=57, srv793001885=94, srv1093822065=5, srv266879293=63, srv1872641758=47, srv1891228156=50, srv569556584=82, srv1160714495=11, srv863863394=98, srv2084841018=59, srv2030317289=56, srv354156757=69, srv583677301=83, srv165532480=33, srv1553364882=27, srv618563107=85, srv1119370761=7, srv256702140=62, srv1390043281=21, srv410158971=75, srv118221892=13, srv149595270=22, srv163445875=31} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-14 09:53:20,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-14 09:53:20,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-14 09:53:20,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-14 09:53:20,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-14 09:53:20,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-14 09:53:20,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-14 09:53:20,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-14 09:53:20,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-14 09:53:20,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-14 09:53:20,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-14 09:53:20,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-14 09:53:20,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-14 09:53:20,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-04-14 09:53:20,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-04-14 09:53:20,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-04-14 09:53:20,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-04-14 09:53:20,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-04-14 09:53:20,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-04-14 09:53:20,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-04-14 09:53:20,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-04-14 09:53:20,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-04-14 09:53:20,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-04-14 09:53:20,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-04-14 09:53:20,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-04-14 09:53:20,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-04-14 09:53:20,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-04-14 09:53:20,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-04-14 09:53:20,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-04-14 09:53:20,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-04-14 09:53:20,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-04-14 09:53:20,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-04-14 09:53:20,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-04-14 09:53:20,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-04-14 09:53:20,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-04-14 09:53:20,561 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-04-14 09:53:20,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-04-14 09:53:20,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-04-14 09:53:20,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-04-14 09:53:20,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-04-14 09:53:20,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-04-14 09:53:20,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-04-14 09:53:20,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-04-14 09:53:20,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-04-14 09:53:20,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-04-14 09:53:20,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-04-14 09:53:20,562 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-04-14 09:53:20,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-04-14 09:53:20,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-04-14 09:53:20,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-04-14 09:53:20,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-04-14 09:53:20,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-04-14 09:53:20,563 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-04-14 09:53:20,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-04-14 09:53:20,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-04-14 09:53:20,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-04-14 09:53:20,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-04-14 09:53:20,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-04-14 09:53:20,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-04-14 09:53:20,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-04-14 09:53:20,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-04-14 09:53:20,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-04-14 09:53:20,564 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-04-14 09:53:20,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-04-14 09:53:20,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-04-14 09:53:20,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-04-14 09:53:20,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-04-14 09:53:20,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-04-14 09:53:20,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-04-14 09:53:20,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-04-14 09:53:20,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-04-14 09:53:20,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-04-14 09:53:20,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-04-14 09:53:20,565 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-04-14 09:53:20,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-04-14 09:53:20,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-04-14 09:53:20,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-04-14 09:53:20,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-04-14 09:53:20,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-04-14 09:53:20,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-04-14 09:53:20,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-04-14 09:53:20,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-04-14 09:53:20,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-04-14 09:53:20,566 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-04-14 09:53:20,673 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.734519624614819E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-14 09:53:20,674 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1246840739:83 , srv1390043281:83 , srv1533958490:83 , srv1056669629:84 , srv1119370761:84 , srv1179794874:84 , srv1336281591:84 , srv1536524937:84 , srv1538072058:84 , srv1553364882:84 , srv1558585296:84 , srv1607768741:84 , srv165532480:84 , srv1661630029:84 , srv169609168:84 , srv1822008101:84 , srv1830723803:84 , srv1888499048:84 , srv1962579245:84 , srv1972851627:84 , srv1983461671:84 , srv2053784463:84 , srv2127877831:84 , srv256702140:84 , srv287250097:84 , srv31027378:84 , srv34452873:84 , srv375265666:84 , srv3953202:84 , srv396654860:84 , srv417308407:84 , srv476463117:84 , srv483322064:84 , srv534216578:84 , srv560519836:84 , srv618563107:84 , srv653065429:84 , srv654437225:84 , srv660800570:84 , srv750594714:84 , srv763744911:84 , srv78248640:84 , srv793001885:84 , srv816566455:84 , srv825754633:84 , srv863863394:84 , srv987188367:84 , srv1026917440:85 , srv1033160053:85 , srv1053400175:85 , srv1062928719:85 , srv1093822065:85 , srv1107863976:85 , srv1125962149:85 , srv1135503083:85 , srv1150054220:85 , srv1160714495:85 , srv118221892:85 , srv1222812910:85 , srv1310673558:85 , srv1351005719:85 , srv1388645040:85 , srv149595270:85 , srv1540424751:85 , srv1600661103:85 , srv1675340821:85 , srv168455060:85 , srv1709736655:85 , srv1723951352:85 , srv1773121043:85 , srv178289636:85 , srv181295235:85 , srv1857519778:85 , srv1869120651:85 , srv1872641758:85 , srv1886248326:85 , srv1891228156:85 , srv2018157570:85 , srv2030317289:85 , srv2039614665:85 , srv2115564964:85 , srv32716129:85 , srv337436043:85 , srv354156757:85 , srv369887682:85 , srv376329082:85 , srv410158971:85 , srv564720889:85 , srv569556584:85 , srv583677301:85 , srv611948404:85 , srv677955032:85 , srv68686373:85 , srv794040970:85 , srv1306893056:86 , srv163445875:86 , srv164587798:86 , srv2084841018:86 , srv266879293:86 , srv2004989792:630 }
2024-04-14 09:53:20,711 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=304 (was 304), ProcessCount=169 (was 169), AvailableMemoryMB=7865 (was 8613)
2024-04-14 09:53:20,722 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=304, ProcessCount=169, AvailableMemoryMB=7865
2024-04-14 09:53:20,722 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-04-14 09:53:20,723 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-04-14 09:53:20,733 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1035880238:84 , srv1066809503:84 , srv1108433564:84 , srv1108644669:84 , srv1221405009:84 , srv1239105384:84 , srv127927371:84 , srv1283433730:84 , srv129324237:84 , srv1387407838:84 , srv1406028610:84 , srv1414624582:84 , srv1444316977:84 , srv1512710810:84 , srv1595357606:84 , srv1695324815:84 , srv1809928931:84 , srv248993813:84 , srv287809448:84 , srv319492040:84 , srv40668316:84 , srv416692114:84 , srv435289383:84 , srv510546980:84 , srv683877285:84 , srv765056926:84 , srv781210024:84 , srv840776245:84 , srv884448814:84 , srv762186020:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-04-14 09:53:20,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-14 09:53:20,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv510546980=23, srv1283433730=7, srv127927371=6, srv1414624582=11, srv1595357606=14, srv1066809503=1, srv762186020=25, srv40668316=20, srv683877285=24, srv287809448=18, srv129324237=8, srv416692114=21, srv1108644669=3, srv1035880238=0, srv1406028610=10, srv1809928931=16, srv1387407838=9, srv840776245=28, srv765056926=26, srv1221405009=4, srv1512710810=13, srv1444316977=12, srv248993813=17, srv884448814=29, srv1695324815=15, srv319492040=19, srv435289383=22, srv781210024=27, srv1108433564=2, srv1239105384=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-14 09:53:20,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-14 09:53:20,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-14 09:53:20,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-14 09:53:20,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-14 09:53:20,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-14 09:53:20,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-14 09:53:20,744 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-14 09:53:20,745 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-14 09:53:20,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-14 09:53:20,746 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-04-14 09:53:20,763 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1227493968248639, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-14 09:53:20,764 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912579822605538, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1227493968248639, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-04-14 09:55:20,749 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6254291 different iterations.  Found a solution that moves 1849 regions; Going from a computed imbalance of 0.9912579822605538 to a new imbalance of 4.3156582404131396E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6848148148148148, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-14 09:55:20,757 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1035880238:90 , srv1066809503:90 , srv1108433564:90 , srv1108644669:90 , srv1221405009:90 , srv1239105384:90 , srv127927371:90 , srv1283433730:90 , srv129324237:90 , srv1387407838:90 , srv1406028610:90 , srv1414624582:90 , srv1444316977:90 , srv1512710810:90 , srv1595357606:90 , srv1695324815:90 , srv1809928931:90 , srv248993813:90 , srv287809448:90 , srv319492040:90 , srv40668316:90 , srv416692114:90 , srv435289383:90 , srv510546980:90 , srv683877285:90 , srv762186020:90 , srv765056926:90 , srv781210024:90 , srv840776245:90 , srv884448814:90 }
2024-04-14 09:55:20,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-04-14 09:55:20,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv510546980=23, srv1283433730=7, srv127927371=6, srv1414624582=11, srv1595357606=14, srv1066809503=1, srv762186020=25, srv40668316=20, srv683877285=24, srv287809448=18, srv129324237=8, srv416692114=21, srv1108644669=3, srv1035880238=0, srv1406028610=10, srv1809928931=16, srv1387407838=9, srv840776245=28, srv765056926=26, srv1221405009=4, srv1512710810=13, srv1444316977=12, srv248993813=17, srv884448814=29, srv1695324815=15, srv319492040=19, srv435289383=22, srv781210024=27, srv1108433564=2, srv1239105384=5} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-04-14 09:55:20,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-04-14 09:55:20,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-04-14 09:55:20,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-04-14 09:55:20,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-04-14 09:55:20,762 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-04-14 09:55:20,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-04-14 09:55:20,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-04-14 09:55:20,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-04-14 09:55:20,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-04-14 09:55:20,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-04-14 09:55:20,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-04-14 09:55:20,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-04-14 09:55:20,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-04-14 09:55:20,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-04-14 09:55:20,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-04-14 09:55:20,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-04-14 09:55:20,763 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-04-14 09:55:20,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-04-14 09:55:20,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-04-14 09:55:20,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-04-14 09:55:20,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-04-14 09:55:20,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-04-14 09:55:20,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-04-14 09:55:20,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-04-14 09:55:20,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-04-14 09:55:20,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-04-14 09:55:20,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-04-14 09:55:20,764 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-04-14 09:55:20,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-04-14 09:55:20,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-04-14 09:55:20,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-04-14 09:55:20,782 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-04-14 09:55:20,783 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1035880238:90 , srv1066809503:90 , srv1108433564:90 , srv1108644669:90 , srv1221405009:90 , srv1239105384:90 , srv127927371:90 , srv1283433730:90 , srv129324237:90 , srv1387407838:90 , srv1406028610:90 , srv1414624582:90 , srv1444316977:90 , srv1512710810:90 , srv1595357606:90 , srv1695324815:90 , srv1809928931:90 , srv248993813:90 , srv287809448:90 , srv319492040:90 , srv40668316:90 , srv416692114:90 , srv435289383:90 , srv510546980:90 , srv683877285:90 , srv762186020:90 , srv765056926:90 , srv781210024:90 , srv840776245:90 , srv884448814:90 }
2024-04-14 09:55:20,797 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=246 (was 304), ProcessCount=167 (was 169), AvailableMemoryMB=7100 (was 7865)