2023-03-31 11:55:56,275 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-31 11:55:56,696 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-31 11:55:56,838 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-31 11:55:56,911 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-31 11:55:56,921 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-31 11:55:56,939 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=384, ProcessCount=169, AvailableMemoryMB=4162
2023-03-31 11:55:56,941 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-31 11:55:56,941 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-31 11:55:57,092 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1002573307:84 , srv1048114365:84 , srv1069760661:84 , srv1095851891:84 , srv1123054493:84 , srv1139153095:84 , srv115020822:84 , srv1166718099:84 , srv1185147369:84 , srv1199301876:84 , srv1201744032:84 , srv1225159431:84 , srv1233306505:84 , srv1257609816:84 , srv1259143677:84 , srv1283916313:84 , srv1294921002:84 , srv1339733473:84 , srv1373307192:84 , srv1394948439:84 , srv1400817442:84 , srv1410563282:84 , srv1418005328:84 , srv1419275389:84 , srv1442774165:84 , srv1468054927:84 , srv1469889684:84 , srv1532984947:84 , srv1533528730:84 , srv1552478365:84 , srv1620358224:84 , srv1672468957:84 , srv168453191:84 , srv1688294859:84 , srv1688306607:84 , srv1700323404:84 , srv1714651845:84 , srv1732329461:84 , srv1750096845:84 , srv175891449:84 , srv1771185402:84 , srv1804435643:84 , srv1834858741:84 , srv1886695549:84 , srv1888556314:84 , srv189546521:84 , srv1905362016:84 , srv1924560322:84 , srv1929668628:84 , srv1940636423:84 , srv1955746473:84 , srv1962790856:84 , srv1965016327:84 , srv2033551793:84 , srv2051543519:84 , srv205407175:84 , srv2061895407:84 , srv2087207766:84 , srv2117367628:84 , srv2143364422:84 , srv224967163:84 , srv243591745:84 , srv247534607:84 , srv26701856:84 , srv269505359:84 , srv302298153:84 , srv302301582:84 , srv363210014:84 , srv365422115:84 , srv36862003:84 , srv382862960:84 , srv416718088:84 , srv490562102:84 , srv496400964:84 , srv499105865:84 , srv531771621:84 , srv536219725:84 , srv544697792:84 , srv54687605:84 , srv597169296:84 , srv63822044:84 , srv711826526:84 , srv740439491:84 , srv767497773:84 , srv779473098:84 , srv785239741:84 , srv792728479:84 , srv798839434:84 , srv82790440:84 , srv865889439:84 , srv86796639:84 , srv881185863:84 , srv888276801:84 , srv893549840:84 , srv897285277:84 , srv927562264:84 , srv959417621:84 , srv96841771:84 , srv985422390:84 , srv1660491346:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-31 11:55:57,104 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-31 11:55:57,119 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1532984947=27, srv865889439=90, srv1394948439=19, srv959417621=97, srv1700323404=36, srv243591745=62, srv1750096845=39, srv2117367628=59, srv798839434=88, srv1199301876=9, srv1924560322=48, srv1620358224=30, srv205407175=56, srv1225159431=11, srv382862960=71, srv1442774165=24, srv1940636423=50, srv1201744032=10, srv1283916313=15, srv881185863=92, srv1552478365=29, srv189546521=46, srv1660491346=31, srv1419275389=23, srv1688306607=35, srv2087207766=58, srv1688294859=34, srv1166718099=7, srv1804435643=42, srv1714651845=37, srv893549840=94, srv544697792=78, srv499105865=75, srv302298153=66, srv1771185402=41, srv1259143677=14, srv168453191=33, srv363210014=68, srv365422115=69, srv224967163=61, srv785239741=86, srv927562264=96, srv247534607=63, srv1469889684=26, srv1257609816=13, srv779473098=85, srv26701856=64, srv302301582=67, srv1069760661=2, srv175891449=40, srv2033551793=54, srv54687605=79, srv531771621=76, srv1373307192=18, srv115020822=6, srv86796639=91, srv82790440=89, srv1962790856=52, srv2051543519=55, srv888276801=93, srv490562102=73, srv597169296=80, srv1834858741=43, srv1965016327=53, srv1732329461=38, srv1955746473=51, srv792728479=87, srv1468054927=25, srv269505359=65, srv1123054493=4, srv711826526=82, srv2061895407=57, srv1048114365=1, srv767497773=84, srv1095851891=3, srv1905362016=47, srv1929668628=49, srv740439491=83, srv985422390=99, srv36862003=70, srv1139153095=5, srv897285277=95, srv536219725=77, srv1233306505=12, srv1533528730=28, srv1418005328=22, srv1886695549=44, srv1672468957=32, srv2143364422=60, srv1185147369=8, srv1888556314=45, srv416718088=72, srv96841771=98, srv1002573307=0, srv1339733473=17, srv63822044=81, srv496400964=74, srv1410563282=21, srv1400817442=20, srv1294921002=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-31 11:55:57,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-31 11:55:57,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-31 11:55:57,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-31 11:55:57,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-31 11:55:57,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-31 11:55:57,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-31 11:55:57,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-31 11:55:57,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-31 11:55:57,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-31 11:55:57,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-31 11:55:57,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-31 11:55:57,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-31 11:55:57,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-31 11:55:57,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-31 11:55:57,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-31 11:55:57,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-31 11:55:57,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-31 11:55:57,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-31 11:55:57,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-31 11:55:57,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-31 11:55:57,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-31 11:55:57,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-31 11:55:57,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-31 11:55:57,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-31 11:55:57,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-31 11:55:57,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-31 11:55:57,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-31 11:55:57,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-31 11:55:57,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-31 11:55:57,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-31 11:55:57,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-31 11:55:57,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-31 11:55:57,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-31 11:55:57,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-31 11:55:57,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-31 11:55:57,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-31 11:55:57,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-31 11:55:57,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-31 11:55:57,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-31 11:55:57,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-31 11:55:57,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-31 11:55:57,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-31 11:55:57,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-31 11:55:57,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-31 11:55:57,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-31 11:55:57,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-31 11:55:57,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-31 11:55:57,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-31 11:55:57,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-31 11:55:57,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-31 11:55:57,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-31 11:55:57,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-31 11:55:57,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-31 11:55:57,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-31 11:55:57,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-31 11:55:57,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-31 11:55:57,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-31 11:55:57,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-31 11:55:57,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-31 11:55:57,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-31 11:55:57,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-31 11:55:57,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-31 11:55:57,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-31 11:55:57,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-31 11:55:57,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-31 11:55:57,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-31 11:55:57,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-31 11:55:57,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-31 11:55:57,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-31 11:55:57,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-31 11:55:57,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-31 11:55:57,265 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-31 11:55:57,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-31 11:55:57,267 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-31 11:55:57,268 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-31 11:55:57,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-31 11:55:57,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-31 11:55:57,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-31 11:55:57,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-31 11:55:57,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-31 11:55:57,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-31 11:55:57,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-31 11:55:57,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-31 11:55:57,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-31 11:55:57,270 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-31 11:55:57,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-31 11:55:57,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-31 11:55:57,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-31 11:55:57,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-31 11:55:57,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-31 11:55:57,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-31 11:55:57,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-31 11:55:57,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-31 11:55:57,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-31 11:55:57,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-31 11:55:57,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-31 11:55:57,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-31 11:55:57,272 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-31 11:55:57,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-31 11:55:57,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-31 11:55:57,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-31 11:55:57,520 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-31 11:55:57,521 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-31 11:55:57,521 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-31 11:55:57,924 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 610 ms to try 5000 different iterations.  Found a solution that moves 4675 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17946892993112634. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03970586455751069, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05998316262068713, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5194444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03970586455751069, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24633333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-31 11:55:57,944 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv115020822:82 , srv1620358224:82 , srv1732329461:82 , srv597169296:82 , srv959417621:82 , srv1139153095:83 , srv1700323404:83 , srv1771185402:83 , srv1069760661:84 , srv1166718099:84 , srv1199301876:84 , srv1339733473:84 , srv1469889684:84 , srv1552478365:84 , srv1940636423:84 , srv2051543519:84 , srv2117367628:84 , srv302298153:84 , srv536219725:84 , srv86796639:84 , srv1201744032:85 , srv1257609816:85 , srv1259143677:85 , srv1442774165:85 , srv1714651845:85 , srv1804435643:85 , srv1834858741:85 , srv1888556314:85 , srv189546521:85 , srv1929668628:85 , srv205407175:85 , srv2143364422:85 , srv243591745:85 , srv779473098:85 , srv785239741:85 , srv792728479:85 , srv865889439:85 , srv893549840:85 , srv1002573307:86 , srv1233306505:86 , srv1283916313:86 , srv1373307192:86 , srv1410563282:86 , srv1886695549:86 , srv1955746473:86 , srv2033551793:86 , srv363210014:86 , srv36862003:86 , srv496400964:86 , srv54687605:86 , srv63822044:86 , srv711826526:86 , srv82790440:86 , srv881185863:86 , srv927562264:86 , srv96841771:86 , srv985422390:86 , srv1048114365:87 , srv1225159431:87 , srv1468054927:87 , srv168453191:87 , srv1688294859:87 , srv302301582:87 , srv365422115:87 , srv416718088:87 , srv544697792:87 , srv740439491:87 , srv1123054493:88 , srv1400817442:88 , srv1419275389:88 , srv1750096845:88 , srv1905362016:88 , srv1924560322:88 , srv224967163:88 , srv269505359:88 , srv382862960:88 , srv531771621:88 , srv767497773:88 , srv888276801:88 , srv1095851891:89 , srv1185147369:89 , srv1294921002:89 , srv1394948439:89 , srv1418005328:89 , srv175891449:89 , srv1965016327:89 , srv490562102:89 , srv1532984947:90 , srv1533528730:90 , srv1672468957:90 , srv1688306607:90 , srv1962790856:90 , srv247534607:90 , srv499105865:90 , srv2061895407:91 , srv26701856:91 , srv798839434:91 , srv2087207766:92 , srv897285277:93 , srv1660491346:443 }
2023-03-31 11:55:57,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-31 11:55:57,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1532984947=27, srv865889439=90, srv1394948439=19, srv959417621=97, srv1700323404=36, srv243591745=62, srv1750096845=39, srv2117367628=59, srv798839434=88, srv1199301876=9, srv1924560322=48, srv1620358224=30, srv205407175=56, srv1225159431=11, srv382862960=71, srv1442774165=24, srv1940636423=50, srv1201744032=10, srv1283916313=15, srv881185863=92, srv1552478365=29, srv189546521=46, srv1660491346=31, srv1419275389=23, srv1688306607=35, srv2087207766=58, srv1688294859=34, srv1166718099=7, srv1804435643=42, srv1714651845=37, srv893549840=94, srv544697792=78, srv499105865=75, srv302298153=66, srv1771185402=41, srv1259143677=14, srv168453191=33, srv363210014=68, srv365422115=69, srv224967163=61, srv785239741=86, srv927562264=96, srv247534607=63, srv1469889684=26, srv1257609816=13, srv779473098=85, srv26701856=64, srv302301582=67, srv1069760661=2, srv175891449=40, srv2033551793=54, srv54687605=79, srv531771621=76, srv1373307192=18, srv115020822=6, srv86796639=91, srv82790440=89, srv1962790856=52, srv2051543519=55, srv888276801=93, srv490562102=73, srv597169296=80, srv1834858741=43, srv1965016327=53, srv1732329461=38, srv1955746473=51, srv792728479=87, srv1468054927=25, srv269505359=65, srv1123054493=4, srv711826526=82, srv2061895407=57, srv1048114365=1, srv767497773=84, srv1095851891=3, srv1905362016=47, srv1929668628=49, srv740439491=83, srv985422390=99, srv36862003=70, srv1139153095=5, srv897285277=95, srv536219725=77, srv1233306505=12, srv1533528730=28, srv1418005328=22, srv1886695549=44, srv1672468957=32, srv2143364422=60, srv1185147369=8, srv1888556314=45, srv416718088=72, srv96841771=98, srv1002573307=0, srv1339733473=17, srv63822044=81, srv496400964=74, srv1410563282=21, srv1400817442=20, srv1294921002=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-31 11:55:57,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-31 11:55:57,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-31 11:55:57,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-31 11:55:57,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-31 11:55:57,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-31 11:55:57,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-31 11:55:57,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-31 11:55:57,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-31 11:55:57,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-31 11:55:57,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-31 11:55:57,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-31 11:55:57,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-31 11:55:57,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-31 11:55:57,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-31 11:55:57,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-31 11:55:57,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-31 11:55:57,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-31 11:55:57,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-31 11:55:57,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-31 11:55:57,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-31 11:55:57,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-31 11:55:57,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-31 11:55:57,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-31 11:55:57,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-31 11:55:57,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-31 11:55:57,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-31 11:55:57,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-31 11:55:57,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-31 11:55:57,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-31 11:55:57,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-31 11:55:57,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-31 11:55:57,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-31 11:55:57,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-31 11:55:57,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-31 11:55:57,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-31 11:55:57,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-31 11:55:57,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-31 11:55:57,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-31 11:55:57,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-31 11:55:57,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-31 11:55:57,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-31 11:55:57,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-31 11:55:57,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-31 11:55:57,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-31 11:55:57,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-31 11:55:57,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-31 11:55:57,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-31 11:55:57,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-31 11:55:57,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-31 11:55:57,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-31 11:55:57,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-31 11:55:57,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-31 11:55:57,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-31 11:55:57,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-31 11:55:57,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-31 11:55:57,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-31 11:55:57,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-31 11:55:57,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-31 11:55:57,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-31 11:55:57,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-31 11:55:57,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-31 11:55:57,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-31 11:55:57,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-31 11:55:57,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-31 11:55:57,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-31 11:55:57,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-31 11:55:57,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-31 11:55:57,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-31 11:55:57,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-31 11:55:57,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-31 11:55:57,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-31 11:55:57,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-31 11:55:57,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-31 11:55:57,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-31 11:55:57,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-31 11:55:57,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-31 11:55:57,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-31 11:55:57,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-31 11:55:57,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-31 11:55:57,993 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-31 11:55:57,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-31 11:55:57,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-31 11:55:57,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-31 11:55:57,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-31 11:55:57,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-31 11:55:57,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-31 11:55:57,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-31 11:55:57,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-31 11:55:57,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-31 11:55:57,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-31 11:55:57,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-31 11:55:57,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-31 11:55:57,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-31 11:55:57,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-31 11:55:57,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-31 11:55:57,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-31 11:55:57,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-31 11:55:57,995 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-31 11:55:57,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-31 11:55:57,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-31 11:55:57,996 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-31 11:55:58,175 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03970586455751069, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05998316262068713, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03970586455751069, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24633333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-31 11:55:58,175 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-31 11:55:58,176 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17943619488146612, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03970586455751069, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05998316262068713, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03970586455751069, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17416666666666666, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24633333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-31 11:55:58,614 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 585 ms to try 5000 different iterations.  Found a solution that moves 2719 regions; Going from a computed imbalance of 0.17943619488146612 to a new imbalance of 0.01500991086358237. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020034264201923734); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03717885306999732, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.3021111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020034264201923734); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.049416666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-31 11:55:58,632 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2087207766:78 , srv897285277:80 , srv1394948439:81 , srv1418005328:81 , srv1962790856:81 , srv798839434:81 , srv1095851891:82 , srv1419275389:82 , srv1672468957:82 , srv26701856:82 , srv1294921002:83 , srv1532984947:83 , srv1688306607:83 , srv1965016327:83 , srv2061895407:83 , srv224967163:83 , srv247534607:83 , srv490562102:83 , srv767497773:83 , srv1123054493:84 , srv1185147369:84 , srv1225159431:84 , srv1233306505:84 , srv1400817442:84 , srv1468054927:84 , srv1533528730:84 , srv175891449:84 , srv2033551793:84 , srv269505359:84 , srv382862960:84 , srv416718088:84 , srv499105865:84 , srv740439491:84 , srv888276801:84 , srv1002573307:85 , srv1048114365:85 , srv1410563282:85 , srv168453191:85 , srv1924560322:85 , srv205407175:85 , srv243591745:85 , srv496400964:85 , srv531771621:85 , srv63822044:85 , srv96841771:85 , srv1688294859:86 , srv1714651845:86 , srv1750096845:86 , srv1905362016:86 , srv1955746473:86 , srv2143364422:86 , srv302301582:86 , srv363210014:86 , srv792728479:86 , srv82790440:86 , srv1069760661:87 , srv1201744032:87 , srv1257609816:87 , srv1283916313:87 , srv1373307192:87 , srv1552478365:87 , srv1834858741:87 , srv1886695549:87 , srv1888556314:87 , srv189546521:87 , srv1929668628:87 , srv1940636423:87 , srv2051543519:87 , srv302298153:87 , srv36862003:87 , srv544697792:87 , srv785239741:87 , srv86796639:87 , srv881185863:87 , srv927562264:87 , srv115020822:88 , srv1259143677:88 , srv1339733473:88 , srv1469889684:88 , srv1620358224:88 , srv365422115:88 , srv536219725:88 , srv54687605:88 , srv779473098:88 , srv893549840:88 , srv985422390:88 , srv1166718099:89 , srv1804435643:89 , srv2117367628:89 , srv711826526:89 , srv1139153095:90 , srv1199301876:90 , srv1732329461:90 , srv1771185402:90 , srv865889439:90 , srv959417621:90 , srv1442774165:91 , srv597169296:91 , srv1700323404:93 , srv1660491346:509 }
2023-03-31 11:55:58,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-31 11:55:58,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1532984947=27, srv865889439=90, srv1394948439=19, srv959417621=97, srv1700323404=36, srv243591745=62, srv1750096845=39, srv2117367628=59, srv798839434=88, srv1199301876=9, srv1924560322=48, srv1620358224=30, srv205407175=56, srv1225159431=11, srv382862960=71, srv1442774165=24, srv1940636423=50, srv1201744032=10, srv1283916313=15, srv881185863=92, srv1552478365=29, srv189546521=46, srv1660491346=31, srv1419275389=23, srv1688306607=35, srv2087207766=58, srv1688294859=34, srv1166718099=7, srv1804435643=42, srv1714651845=37, srv893549840=94, srv544697792=78, srv499105865=75, srv302298153=66, srv1771185402=41, srv1259143677=14, srv168453191=33, srv363210014=68, srv365422115=69, srv224967163=61, srv785239741=86, srv927562264=96, srv247534607=63, srv1469889684=26, srv1257609816=13, srv779473098=85, srv26701856=64, srv302301582=67, srv1069760661=2, srv175891449=40, srv2033551793=54, srv54687605=79, srv531771621=76, srv1373307192=18, srv115020822=6, srv86796639=91, srv82790440=89, srv1962790856=52, srv2051543519=55, srv888276801=93, srv490562102=73, srv597169296=80, srv1834858741=43, srv1965016327=53, srv1732329461=38, srv1955746473=51, srv792728479=87, srv1468054927=25, srv269505359=65, srv1123054493=4, srv711826526=82, srv2061895407=57, srv1048114365=1, srv767497773=84, srv1095851891=3, srv1905362016=47, srv1929668628=49, srv740439491=83, srv985422390=99, srv36862003=70, srv1139153095=5, srv897285277=95, srv536219725=77, srv1233306505=12, srv1533528730=28, srv1418005328=22, srv1886695549=44, srv1672468957=32, srv2143364422=60, srv1185147369=8, srv1888556314=45, srv416718088=72, srv96841771=98, srv1002573307=0, srv1339733473=17, srv63822044=81, srv496400964=74, srv1410563282=21, srv1400817442=20, srv1294921002=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-31 11:55:58,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-31 11:55:58,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-31 11:55:58,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-31 11:55:58,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-31 11:55:58,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-31 11:55:58,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-31 11:55:58,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-31 11:55:58,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-31 11:55:58,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-31 11:55:58,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-31 11:55:58,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-31 11:55:58,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-31 11:55:58,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-31 11:55:58,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-31 11:55:58,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-31 11:55:58,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-31 11:55:58,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-31 11:55:58,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-31 11:55:58,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-31 11:55:58,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-31 11:55:58,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-31 11:55:58,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-31 11:55:58,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-31 11:55:58,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-31 11:55:58,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-31 11:55:58,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-31 11:55:58,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-31 11:55:58,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-31 11:55:58,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-31 11:55:58,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-31 11:55:58,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-31 11:55:58,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-31 11:55:58,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-31 11:55:58,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-31 11:55:58,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-31 11:55:58,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-31 11:55:58,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-31 11:55:58,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-31 11:55:58,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-31 11:55:58,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-31 11:55:58,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-31 11:55:58,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-31 11:55:58,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-31 11:55:58,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-31 11:55:58,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-31 11:55:58,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-31 11:55:58,663 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-31 11:55:58,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-31 11:55:58,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-31 11:55:58,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-31 11:55:58,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-31 11:55:58,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-31 11:55:58,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-31 11:55:58,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-31 11:55:58,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-31 11:55:58,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-31 11:55:58,664 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-31 11:55:58,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-31 11:55:58,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-31 11:55:58,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-31 11:55:58,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-31 11:55:58,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-31 11:55:58,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-31 11:55:58,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-31 11:55:58,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-31 11:55:58,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-31 11:55:58,665 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-31 11:55:58,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-31 11:55:58,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-31 11:55:58,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-31 11:55:58,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-31 11:55:58,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-31 11:55:58,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-31 11:55:58,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-31 11:55:58,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-31 11:55:58,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-31 11:55:58,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-31 11:55:58,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-31 11:55:58,666 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-31 11:55:58,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-31 11:55:58,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-31 11:55:58,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-31 11:55:58,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-31 11:55:58,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-31 11:55:58,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-31 11:55:58,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-31 11:55:58,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-31 11:55:58,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-31 11:55:58,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-31 11:55:58,667 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-31 11:55:58,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-31 11:55:58,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-31 11:55:58,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-31 11:55:58,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-31 11:55:58,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-31 11:55:58,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-31 11:55:58,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-31 11:55:58,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-31 11:55:58,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-31 11:55:58,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-31 11:55:58,668 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-31 11:55:58,826 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020034264201923734); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03717885306999732, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020034264201923734); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.049416666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-31 11:55:58,826 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-31 11:55:58,826 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014990872018656978, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.020034264201923734); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03717885306999732, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.020034264201923734); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.049416666666666664, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-31 11:55:59,216 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 532 ms to try 5000 different iterations.  Found a solution that moves 1048 regions; Going from a computed imbalance of 0.014990872018656978 to a new imbalance of 7.040713102560903E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005691928901547949); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010358338378419612); MoveCostFunction : (multiplier=7.0, imbalance=0.11644444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005691928901547949); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-31 11:55:59,218 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1700323404:80 , srv1442774165:82 , srv1533528730:82 , srv865889439:82 , srv1166718099:83 , srv1804435643:83 , srv302301582:83 , srv365422115:83 , srv927562264:83 , srv1123054493:84 , srv1199301876:84 , srv1201744032:84 , srv1688306607:84 , srv1771185402:84 , srv1886695549:84 , srv1905362016:84 , srv36862003:84 , srv499105865:84 , srv531771621:84 , srv544697792:84 , srv54687605:84 , srv597169296:84 , srv711826526:84 , srv767497773:84 , srv881185863:84 , srv893549840:84 , srv897285277:84 , srv985422390:84 , srv1139153095:85 , srv1185147369:85 , srv1225159431:85 , srv1259143677:85 , srv1283916313:85 , srv1294921002:85 , srv1339733473:85 , srv1373307192:85 , srv1400817442:85 , srv1469889684:85 , srv1532984947:85 , srv1688294859:85 , srv1732329461:85 , srv175891449:85 , srv1834858741:85 , srv1924560322:85 , srv1965016327:85 , srv2061895407:85 , srv2117367628:85 , srv247534607:85 , srv26701856:85 , srv269505359:85 , srv363210014:85 , srv382862960:85 , srv779473098:85 , srv798839434:85 , srv86796639:85 , srv959417621:85 , srv1048114365:86 , srv1069760661:86 , srv1257609816:86 , srv1394948439:86 , srv1552478365:86 , srv1672468957:86 , srv1750096845:86 , srv189546521:86 , srv1929668628:86 , srv1955746473:86 , srv1962790856:86 , srv416718088:86 , srv490562102:86 , srv496400964:86 , srv536219725:86 , srv785239741:86 , srv888276801:86 , srv96841771:86 , srv1002573307:87 , srv1095851891:87 , srv115020822:87 , srv1233306505:87 , srv1410563282:87 , srv1418005328:87 , srv1419275389:87 , srv1468054927:87 , srv1620358224:87 , srv168453191:87 , srv1714651845:87 , srv1888556314:87 , srv1940636423:87 , srv2033551793:87 , srv2051543519:87 , srv205407175:87 , srv2143364422:87 , srv243591745:87 , srv302298153:87 , srv792728479:87 , srv82790440:87 , srv2087207766:88 , srv224967163:88 , srv63822044:88 , srv740439491:88 , srv1660491346:556 }
2023-03-31 11:55:59,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-31 11:55:59,226 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1532984947=27, srv865889439=90, srv1394948439=19, srv959417621=97, srv1700323404=36, srv243591745=62, srv1750096845=39, srv2117367628=59, srv798839434=88, srv1199301876=9, srv1924560322=48, srv1620358224=30, srv205407175=56, srv1225159431=11, srv382862960=71, srv1442774165=24, srv1940636423=50, srv1201744032=10, srv1283916313=15, srv881185863=92, srv1552478365=29, srv189546521=46, srv1660491346=31, srv1419275389=23, srv1688306607=35, srv2087207766=58, srv1688294859=34, srv1166718099=7, srv1804435643=42, srv1714651845=37, srv893549840=94, srv544697792=78, srv499105865=75, srv302298153=66, srv1771185402=41, srv1259143677=14, srv168453191=33, srv363210014=68, srv365422115=69, srv224967163=61, srv785239741=86, srv927562264=96, srv247534607=63, srv1469889684=26, srv1257609816=13, srv779473098=85, srv26701856=64, srv302301582=67, srv1069760661=2, srv175891449=40, srv2033551793=54, srv54687605=79, srv531771621=76, srv1373307192=18, srv115020822=6, srv86796639=91, srv82790440=89, srv1962790856=52, srv2051543519=55, srv888276801=93, srv490562102=73, srv597169296=80, srv1834858741=43, srv1965016327=53, srv1732329461=38, srv1955746473=51, srv792728479=87, srv1468054927=25, srv269505359=65, srv1123054493=4, srv711826526=82, srv2061895407=57, srv1048114365=1, srv767497773=84, srv1095851891=3, srv1905362016=47, srv1929668628=49, srv740439491=83, srv985422390=99, srv36862003=70, srv1139153095=5, srv897285277=95, srv536219725=77, srv1233306505=12, srv1533528730=28, srv1418005328=22, srv1886695549=44, srv1672468957=32, srv2143364422=60, srv1185147369=8, srv1888556314=45, srv416718088=72, srv96841771=98, srv1002573307=0, srv1339733473=17, srv63822044=81, srv496400964=74, srv1410563282=21, srv1400817442=20, srv1294921002=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-31 11:55:59,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-31 11:55:59,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-31 11:55:59,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-31 11:55:59,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-31 11:55:59,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-31 11:55:59,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-31 11:55:59,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-31 11:55:59,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-31 11:55:59,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-31 11:55:59,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-31 11:55:59,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-31 11:55:59,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-31 11:55:59,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-31 11:55:59,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-31 11:55:59,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-31 11:55:59,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-31 11:55:59,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-31 11:55:59,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-31 11:55:59,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-31 11:55:59,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-31 11:55:59,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-31 11:55:59,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-31 11:55:59,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-31 11:55:59,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-31 11:55:59,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-31 11:55:59,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-31 11:55:59,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-31 11:55:59,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-31 11:55:59,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-31 11:55:59,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-31 11:55:59,254 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-31 11:55:59,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-31 11:55:59,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-31 11:55:59,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-31 11:55:59,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-31 11:55:59,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-31 11:55:59,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-31 11:55:59,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-31 11:55:59,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-31 11:55:59,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-31 11:55:59,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-31 11:55:59,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-31 11:55:59,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-31 11:55:59,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-31 11:55:59,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-31 11:55:59,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-31 11:55:59,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-31 11:55:59,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-31 11:55:59,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-31 11:55:59,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-31 11:55:59,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-31 11:55:59,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-31 11:55:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-31 11:55:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-31 11:55:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-31 11:55:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-31 11:55:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-31 11:55:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-31 11:55:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-31 11:55:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-31 11:55:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-31 11:55:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-31 11:55:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-31 11:55:59,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-31 11:55:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-31 11:55:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-31 11:55:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-31 11:55:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-31 11:55:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-31 11:55:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-31 11:55:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-31 11:55:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-31 11:55:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-31 11:55:59,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-31 11:55:59,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-31 11:55:59,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-31 11:55:59,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-31 11:55:59,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-31 11:55:59,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-31 11:55:59,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-31 11:55:59,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-31 11:55:59,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-31 11:55:59,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-31 11:55:59,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-31 11:55:59,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-31 11:55:59,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-31 11:55:59,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-31 11:55:59,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-31 11:55:59,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-31 11:55:59,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-31 11:55:59,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-31 11:55:59,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-31 11:55:59,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-31 11:55:59,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-31 11:55:59,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-31 11:55:59,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-31 11:55:59,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-31 11:55:59,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-31 11:55:59,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-31 11:55:59,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-31 11:55:59,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-31 11:55:59,431 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005691928901547949); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010358338378419612); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005691928901547949); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-31 11:55:59,431 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-31 11:55:59,431 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.967330574124673E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005691928901547949); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010358338378419612); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005691928901547949); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-31 11:55:59,807 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 528 ms to try 5000 different iterations.  Found a solution that moves 263 regions; Going from a computed imbalance of 6.967330574124673E-4 to a new imbalance of 2.8011786179539255E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.029222222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-31 11:55:59,809 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv168453191:83 , srv1750096845:83 , srv1888556314:83 , srv1955746473:83 , srv2061895407:83 , srv2117367628:83 , srv224967163:83 , srv26701856:83 , srv54687605:83 , srv711826526:83 , srv740439491:83 , srv82790440:83 , srv985422390:83 , srv1002573307:84 , srv1048114365:84 , srv1095851891:84 , srv1139153095:84 , srv1185147369:84 , srv1257609816:84 , srv1259143677:84 , srv1283916313:84 , srv1373307192:84 , srv1410563282:84 , srv1442774165:84 , srv1468054927:84 , srv1532984947:84 , srv1672468957:84 , srv1688294859:84 , srv1714651845:84 , srv1834858741:84 , srv189546521:84 , srv1905362016:84 , srv1924560322:84 , srv1929668628:84 , srv2087207766:84 , srv2143364422:84 , srv247534607:84 , srv302298153:84 , srv365422115:84 , srv490562102:84 , srv499105865:84 , srv544697792:84 , srv63822044:84 , srv779473098:84 , srv785239741:84 , srv792728479:84 , srv897285277:84 , srv1069760661:85 , srv115020822:85 , srv1199301876:85 , srv1201744032:85 , srv1233306505:85 , srv1294921002:85 , srv1339733473:85 , srv1400817442:85 , srv1418005328:85 , srv1419275389:85 , srv1469889684:85 , srv1552478365:85 , srv1620358224:85 , srv1688306607:85 , srv1732329461:85 , srv175891449:85 , srv1771185402:85 , srv1804435643:85 , srv1886695549:85 , srv1940636423:85 , srv1962790856:85 , srv1965016327:85 , srv2033551793:85 , srv2051543519:85 , srv205407175:85 , srv243591745:85 , srv302301582:85 , srv363210014:85 , srv36862003:85 , srv382862960:85 , srv416718088:85 , srv531771621:85 , srv536219725:85 , srv597169296:85 , srv798839434:85 , srv86796639:85 , srv881185863:85 , srv888276801:85 , srv893549840:85 , srv959417621:85 , srv1123054493:86 , srv1166718099:86 , srv1225159431:86 , srv1394948439:86 , srv1533528730:86 , srv1700323404:86 , srv269505359:86 , srv496400964:86 , srv767497773:86 , srv865889439:86 , srv927562264:86 , srv96841771:86 , srv1660491346:633 }
2023-03-31 11:55:59,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-31 11:55:59,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1532984947=27, srv865889439=90, srv1394948439=19, srv959417621=97, srv1700323404=36, srv243591745=62, srv1750096845=39, srv2117367628=59, srv798839434=88, srv1199301876=9, srv1924560322=48, srv1620358224=30, srv205407175=56, srv1225159431=11, srv382862960=71, srv1442774165=24, srv1940636423=50, srv1201744032=10, srv1283916313=15, srv881185863=92, srv1552478365=29, srv189546521=46, srv1660491346=31, srv1419275389=23, srv1688306607=35, srv2087207766=58, srv1688294859=34, srv1166718099=7, srv1804435643=42, srv1714651845=37, srv893549840=94, srv544697792=78, srv499105865=75, srv302298153=66, srv1771185402=41, srv1259143677=14, srv168453191=33, srv363210014=68, srv365422115=69, srv224967163=61, srv785239741=86, srv927562264=96, srv247534607=63, srv1469889684=26, srv1257609816=13, srv779473098=85, srv26701856=64, srv302301582=67, srv1069760661=2, srv175891449=40, srv2033551793=54, srv54687605=79, srv531771621=76, srv1373307192=18, srv115020822=6, srv86796639=91, srv82790440=89, srv1962790856=52, srv2051543519=55, srv888276801=93, srv490562102=73, srv597169296=80, srv1834858741=43, srv1965016327=53, srv1732329461=38, srv1955746473=51, srv792728479=87, srv1468054927=25, srv269505359=65, srv1123054493=4, srv711826526=82, srv2061895407=57, srv1048114365=1, srv767497773=84, srv1095851891=3, srv1905362016=47, srv1929668628=49, srv740439491=83, srv985422390=99, srv36862003=70, srv1139153095=5, srv897285277=95, srv536219725=77, srv1233306505=12, srv1533528730=28, srv1418005328=22, srv1886695549=44, srv1672468957=32, srv2143364422=60, srv1185147369=8, srv1888556314=45, srv416718088=72, srv96841771=98, srv1002573307=0, srv1339733473=17, srv63822044=81, srv496400964=74, srv1410563282=21, srv1400817442=20, srv1294921002=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-31 11:55:59,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-31 11:55:59,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-31 11:55:59,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-31 11:55:59,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-31 11:55:59,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-31 11:55:59,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-31 11:55:59,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-31 11:55:59,829 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-31 11:55:59,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-31 11:55:59,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-31 11:55:59,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-31 11:55:59,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-31 11:55:59,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-31 11:55:59,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-31 11:55:59,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-31 11:55:59,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-31 11:55:59,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-31 11:55:59,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-31 11:55:59,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-31 11:55:59,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-31 11:55:59,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-31 11:55:59,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-31 11:55:59,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-31 11:55:59,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-31 11:55:59,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-31 11:55:59,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-31 11:55:59,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-31 11:55:59,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-31 11:55:59,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-31 11:55:59,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-31 11:55:59,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-31 11:55:59,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-31 11:55:59,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-31 11:55:59,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-31 11:55:59,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-31 11:55:59,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-31 11:55:59,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-31 11:55:59,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-31 11:55:59,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-31 11:55:59,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-31 11:55:59,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-31 11:55:59,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-31 11:55:59,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-31 11:55:59,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-31 11:55:59,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-31 11:55:59,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-31 11:55:59,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-31 11:55:59,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-31 11:55:59,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-31 11:55:59,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-31 11:55:59,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-31 11:55:59,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-31 11:55:59,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-31 11:55:59,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-31 11:55:59,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-31 11:55:59,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-31 11:55:59,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-31 11:55:59,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-31 11:55:59,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-31 11:55:59,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-31 11:55:59,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-31 11:55:59,981 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-31 11:55:59,981 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-31 11:55:59,981 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=2.617022081897356E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018951174787165373); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.6666666666666666E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-31 11:56:00,334 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 486 ms to try 5000 different iterations.  Found a solution that moves 30 regions; Going from a computed imbalance of 2.617022081897356E-5 to a new imbalance of 7.283303786127816E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0033333333333333335); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-31 11:56:00,335 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1199301876:83 , srv1750096845:83 , srv175891449:83 , srv1940636423:83 , srv2051543519:83 , srv269505359:83 , srv496400964:83 , srv536219725:83 , srv63822044:83 , srv865889439:83 , srv888276801:83 , srv96841771:83 , srv1002573307:84 , srv1048114365:84 , srv1069760661:84 , srv1095851891:84 , srv1123054493:84 , srv1139153095:84 , srv115020822:84 , srv1166718099:84 , srv1185147369:84 , srv1225159431:84 , srv1233306505:84 , srv1257609816:84 , srv1259143677:84 , srv1283916313:84 , srv1294921002:84 , srv1339733473:84 , srv1373307192:84 , srv1394948439:84 , srv1400817442:84 , srv1410563282:84 , srv1418005328:84 , srv1419275389:84 , srv1442774165:84 , srv1468054927:84 , srv1469889684:84 , srv1532984947:84 , srv1533528730:84 , srv1552478365:84 , srv1620358224:84 , srv1672468957:84 , srv168453191:84 , srv1688294859:84 , srv1688306607:84 , srv1700323404:84 , srv1714651845:84 , srv1732329461:84 , srv1771185402:84 , srv1804435643:84 , srv1886695549:84 , srv1888556314:84 , srv189546521:84 , srv1905362016:84 , srv1924560322:84 , srv1929668628:84 , srv1962790856:84 , srv1965016327:84 , srv2033551793:84 , srv205407175:84 , srv2061895407:84 , srv2087207766:84 , srv2143364422:84 , srv224967163:84 , srv243591745:84 , srv247534607:84 , srv302298153:84 , srv363210014:84 , srv365422115:84 , srv36862003:84 , srv382862960:84 , srv416718088:84 , srv490562102:84 , srv499105865:84 , srv531771621:84 , srv544697792:84 , srv597169296:84 , srv711826526:84 , srv740439491:84 , srv779473098:84 , srv785239741:84 , srv792728479:84 , srv798839434:84 , srv82790440:84 , srv881185863:84 , srv893549840:84 , srv927562264:84 , srv959417621:84 , srv1201744032:85 , srv1834858741:85 , srv1955746473:85 , srv2117367628:85 , srv26701856:85 , srv302301582:85 , srv54687605:85 , srv767497773:85 , srv86796639:85 , srv897285277:85 , srv985422390:85 , srv1660491346:685 }
2023-03-31 11:56:00,341 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-31 11:56:00,342 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1532984947=27, srv865889439=90, srv1394948439=19, srv959417621=97, srv1700323404=36, srv243591745=62, srv1750096845=39, srv2117367628=59, srv798839434=88, srv1199301876=9, srv1924560322=48, srv1620358224=30, srv205407175=56, srv1225159431=11, srv382862960=71, srv1442774165=24, srv1940636423=50, srv1201744032=10, srv1283916313=15, srv881185863=92, srv1552478365=29, srv189546521=46, srv1660491346=31, srv1419275389=23, srv1688306607=35, srv2087207766=58, srv1688294859=34, srv1166718099=7, srv1804435643=42, srv1714651845=37, srv893549840=94, srv544697792=78, srv499105865=75, srv302298153=66, srv1771185402=41, srv1259143677=14, srv168453191=33, srv363210014=68, srv365422115=69, srv224967163=61, srv785239741=86, srv927562264=96, srv247534607=63, srv1469889684=26, srv1257609816=13, srv779473098=85, srv26701856=64, srv302301582=67, srv1069760661=2, srv175891449=40, srv2033551793=54, srv54687605=79, srv531771621=76, srv1373307192=18, srv115020822=6, srv86796639=91, srv82790440=89, srv1962790856=52, srv2051543519=55, srv888276801=93, srv490562102=73, srv597169296=80, srv1834858741=43, srv1965016327=53, srv1732329461=38, srv1955746473=51, srv792728479=87, srv1468054927=25, srv269505359=65, srv1123054493=4, srv711826526=82, srv2061895407=57, srv1048114365=1, srv767497773=84, srv1095851891=3, srv1905362016=47, srv1929668628=49, srv740439491=83, srv985422390=99, srv36862003=70, srv1139153095=5, srv897285277=95, srv536219725=77, srv1233306505=12, srv1533528730=28, srv1418005328=22, srv1886695549=44, srv1672468957=32, srv2143364422=60, srv1185147369=8, srv1888556314=45, srv416718088=72, srv96841771=98, srv1002573307=0, srv1339733473=17, srv63822044=81, srv496400964=74, srv1410563282=21, srv1400817442=20, srv1294921002=16} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-31 11:56:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-31 11:56:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-31 11:56:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-31 11:56:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-31 11:56:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-31 11:56:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-31 11:56:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-31 11:56:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-31 11:56:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-31 11:56:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-31 11:56:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-31 11:56:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-31 11:56:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-31 11:56:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-31 11:56:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-31 11:56:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-31 11:56:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-31 11:56:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-31 11:56:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-31 11:56:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-31 11:56:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-31 11:56:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-31 11:56:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-31 11:56:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-31 11:56:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-31 11:56:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-31 11:56:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-31 11:56:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-31 11:56:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-31 11:56:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-31 11:56:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-31 11:56:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-31 11:56:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-31 11:56:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-31 11:56:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-31 11:56:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-31 11:56:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-31 11:56:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-31 11:56:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-31 11:56:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-31 11:56:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-31 11:56:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-31 11:56:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-31 11:56:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-31 11:56:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-31 11:56:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-31 11:56:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-31 11:56:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-31 11:56:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-31 11:56:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-31 11:56:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-31 11:56:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-31 11:56:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-31 11:56:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-31 11:56:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-31 11:56:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-31 11:56:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-31 11:56:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-31 11:56:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-31 11:56:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-31 11:56:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-31 11:56:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-31 11:56:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-31 11:56:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-31 11:56:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-31 11:56:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-31 11:56:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-31 11:56:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-31 11:56:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-31 11:56:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-31 11:56:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-31 11:56:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-31 11:56:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-31 11:56:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-31 11:56:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-31 11:56:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-31 11:56:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-31 11:56:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-31 11:56:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-31 11:56:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-31 11:56:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-31 11:56:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-31 11:56:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-31 11:56:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-31 11:56:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-31 11:56:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-31 11:56:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-31 11:56:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-31 11:56:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-31 11:56:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-31 11:56:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-31 11:56:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-31 11:56:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-31 11:56:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-31 11:56:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-31 11:56:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-31 11:56:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-31 11:56:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-31 11:56:00,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-31 11:56:00,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-31 11:56:00,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-31 11:56:00,518 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=7.073239296329447E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-31 11:56:00,518 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1199301876:83 , srv1750096845:83 , srv175891449:83 , srv1940636423:83 , srv2051543519:83 , srv269505359:83 , srv496400964:83 , srv536219725:83 , srv63822044:83 , srv865889439:83 , srv888276801:83 , srv96841771:83 , srv1002573307:84 , srv1048114365:84 , srv1069760661:84 , srv1095851891:84 , srv1123054493:84 , srv1139153095:84 , srv115020822:84 , srv1166718099:84 , srv1185147369:84 , srv1225159431:84 , srv1233306505:84 , srv1257609816:84 , srv1259143677:84 , srv1283916313:84 , srv1294921002:84 , srv1339733473:84 , srv1373307192:84 , srv1394948439:84 , srv1400817442:84 , srv1410563282:84 , srv1418005328:84 , srv1419275389:84 , srv1442774165:84 , srv1468054927:84 , srv1469889684:84 , srv1532984947:84 , srv1533528730:84 , srv1552478365:84 , srv1620358224:84 , srv1672468957:84 , srv168453191:84 , srv1688294859:84 , srv1688306607:84 , srv1700323404:84 , srv1714651845:84 , srv1732329461:84 , srv1771185402:84 , srv1804435643:84 , srv1886695549:84 , srv1888556314:84 , srv189546521:84 , srv1905362016:84 , srv1924560322:84 , srv1929668628:84 , srv1962790856:84 , srv1965016327:84 , srv2033551793:84 , srv205407175:84 , srv2061895407:84 , srv2087207766:84 , srv2143364422:84 , srv224967163:84 , srv243591745:84 , srv247534607:84 , srv302298153:84 , srv363210014:84 , srv365422115:84 , srv36862003:84 , srv382862960:84 , srv416718088:84 , srv490562102:84 , srv499105865:84 , srv531771621:84 , srv544697792:84 , srv597169296:84 , srv711826526:84 , srv740439491:84 , srv779473098:84 , srv785239741:84 , srv792728479:84 , srv798839434:84 , srv82790440:84 , srv881185863:84 , srv893549840:84 , srv927562264:84 , srv959417621:84 , srv1201744032:85 , srv1834858741:85 , srv1955746473:85 , srv2117367628:85 , srv26701856:85 , srv302301582:85 , srv54687605:85 , srv767497773:85 , srv86796639:85 , srv897285277:85 , srv985422390:85 , srv1660491346:685 }
2023-03-31 11:56:00,553 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=417 (was 384) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3683 (was 4162)
2023-03-31 11:56:00,562 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=417, ProcessCount=169, AvailableMemoryMB=3683
2023-03-31 11:56:00,563 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-31 11:56:00,563 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-31 11:56:00,575 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1015894451:84 , srv1022880105:84 , srv1057203744:84 , srv1216674420:84 , srv1374118171:84 , srv1440148462:84 , srv1463277841:84 , srv1510714928:84 , srv1569906702:84 , srv1668404617:84 , srv1741783625:84 , srv1755358733:84 , srv1776170042:84 , srv1819607282:84 , srv1895079729:84 , srv1925408648:84 , srv1938065324:84 , srv1967678641:84 , srv208116065:84 , srv221341555:84 , srv268505619:84 , srv268556133:84 , srv480274133:84 , srv589430861:84 , srv611043501:84 , srv647161082:84 , srv89160292:84 , srv991662218:84 , srv99480712:84 , srv166230260:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-31 11:56:00,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-31 11:56:00,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1216674420=3, srv1015894451=0, srv1755358733=12, srv991662218=28, srv1374118171=4, srv166230260=9, srv1938065324=17, srv99480712=29, srv611043501=25, srv89160292=27, srv1569906702=8, srv1510714928=7, srv1741783625=11, srv221341555=20, srv1463277841=6, srv1440148462=5, srv1895079729=15, srv268505619=21, srv208116065=19, srv1925408648=16, srv1022880105=1, srv647161082=26, srv1776170042=13, srv589430861=24, srv1057203744=2, srv480274133=23, srv1967678641=18, srv268556133=22, srv1668404617=10, srv1819607282=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-31 11:56:00,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-31 11:56:00,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-31 11:56:00,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-31 11:56:00,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-31 11:56:00,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-31 11:56:00,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-31 11:56:00,589 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-31 11:56:00,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-31 11:56:00,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-31 11:56:00,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-31 11:56:00,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-31 11:56:00,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-31 11:56:00,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-31 11:56:00,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-31 11:56:00,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-31 11:56:00,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-31 11:56:00,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-31 11:56:00,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-31 11:56:00,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-31 11:56:00,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-31 11:56:00,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-31 11:56:00,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-31 11:56:00,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-31 11:56:00,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-31 11:56:00,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-31 11:56:00,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-31 11:56:00,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-31 11:56:00,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-31 11:56:00,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-31 11:56:00,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-31 11:56:00,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-31 11:56:00,622 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1188730339132584, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-31 11:56:00,622 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912567608312579, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1188730339132584, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-31 11:58:00,601 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5600731 different iterations.  Found a solution that moves 1864 regions; Going from a computed imbalance of 0.9912567608312579 to a new imbalance of 4.3506689887128686E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6903703703703704, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-31 11:58:00,609 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1015894451:90 , srv1022880105:90 , srv1057203744:90 , srv1216674420:90 , srv1374118171:90 , srv1440148462:90 , srv1463277841:90 , srv1510714928:90 , srv1569906702:90 , srv166230260:90 , srv1668404617:90 , srv1741783625:90 , srv1755358733:90 , srv1776170042:90 , srv1819607282:90 , srv1895079729:90 , srv1925408648:90 , srv1938065324:90 , srv1967678641:90 , srv208116065:90 , srv221341555:90 , srv268505619:90 , srv268556133:90 , srv480274133:90 , srv589430861:90 , srv611043501:90 , srv647161082:90 , srv89160292:90 , srv991662218:90 , srv99480712:90 }
2023-03-31 11:58:00,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-31 11:58:00,610 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1216674420=3, srv1015894451=0, srv1755358733=12, srv991662218=28, srv1374118171=4, srv166230260=9, srv1938065324=17, srv99480712=29, srv611043501=25, srv89160292=27, srv1569906702=8, srv1510714928=7, srv1741783625=11, srv221341555=20, srv1463277841=6, srv1440148462=5, srv1895079729=15, srv268505619=21, srv208116065=19, srv1925408648=16, srv1022880105=1, srv647161082=26, srv1776170042=13, srv589430861=24, srv1057203744=2, srv480274133=23, srv1967678641=18, srv268556133=22, srv1668404617=10, srv1819607282=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-31 11:58:00,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-31 11:58:00,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-31 11:58:00,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-31 11:58:00,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-31 11:58:00,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-31 11:58:00,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-31 11:58:00,613 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-31 11:58:00,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-31 11:58:00,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-31 11:58:00,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-31 11:58:00,632 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-31 11:58:00,632 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1015894451:90 , srv1022880105:90 , srv1057203744:90 , srv1216674420:90 , srv1374118171:90 , srv1440148462:90 , srv1463277841:90 , srv1510714928:90 , srv1569906702:90 , srv166230260:90 , srv1668404617:90 , srv1741783625:90 , srv1755358733:90 , srv1776170042:90 , srv1819607282:90 , srv1895079729:90 , srv1925408648:90 , srv1938065324:90 , srv1967678641:90 , srv208116065:90 , srv221341555:90 , srv268505619:90 , srv268556133:90 , srv480274133:90 , srv589430861:90 , srv611043501:90 , srv647161082:90 , srv89160292:90 , srv991662218:90 , srv99480712:90 }
2023-03-31 11:58:00,647 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=341 (was 417), ProcessCount=168 (was 169), AvailableMemoryMB=4431 (was 3683) - AvailableMemoryMB LEAK? -