2023-04-09 04:01:43,399 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-04-09 04:01:43,659 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-04-09 04:01:43,749 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-04-09 04:01:43,802 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-09 04:01:43,811 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-09 04:01:43,823 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=163, ProcessCount=167, AvailableMemoryMB=4687
2023-04-09 04:01:43,826 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-09 04:01:43,826 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-04-09 04:01:43,946 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1006529008:84 , srv1011754474:84 , srv1033209364:84 , srv1035065683:84 , srv103515921:84 , srv1069301948:84 , srv1088155190:84 , srv1089634862:84 , srv1108413770:84 , srv1123895443:84 , srv1131899228:84 , srv1139537784:84 , srv1141722016:84 , srv1184993849:84 , srv119118112:84 , srv1195293369:84 , srv1197459545:84 , srv1204776373:84 , srv1207725622:84 , srv1221154054:84 , srv1224422342:84 , srv1234471143:84 , srv1256942895:84 , srv1284012166:84 , srv1290494592:84 , srv1310883353:84 , srv1332382164:84 , srv1332654499:84 , srv1342216643:84 , srv1356860474:84 , srv1370558682:84 , srv139474708:84 , srv1406940718:84 , srv1407748336:84 , srv1441082292:84 , srv1453680392:84 , srv1486857011:84 , srv1492924089:84 , srv1506456826:84 , srv1554060389:84 , srv1557945466:84 , srv156810893:84 , srv156995844:84 , srv1570484303:84 , srv1580950724:84 , srv1598814734:84 , srv1600815287:84 , srv1608937401:84 , srv1620915660:84 , srv1644872171:84 , srv1652913697:84 , srv1654965362:84 , srv1658991162:84 , srv1661307226:84 , srv1727301666:84 , srv1738384124:84 , srv1751778448:84 , srv1758164356:84 , srv1830138711:84 , srv1846941659:84 , srv1852123220:84 , srv1858844810:84 , srv1870938921:84 , srv1908353965:84 , srv1914676596:84 , srv1970055917:84 , srv1993918455:84 , srv2012862228:84 , srv2032656077:84 , srv2137645408:84 , srv2139100000:84 , srv219691665:84 , srv232003577:84 , srv313212390:84 , srv315759351:84 , srv338808719:84 , srv365848177:84 , srv372469652:84 , srv428841657:84 , srv461134156:84 , srv499677418:84 , srv523692061:84 , srv524379792:84 , srv541611378:84 , srv56118254:84 , srv57593493:84 , srv649377819:84 , srv676554983:84 , srv716168720:84 , srv752984093:84 , srv77444984:84 , srv7790203:84 , srv852243113:84 , srv884502374:84 , srv894635753:84 , srv934612068:84 , srv950181901:84 , srv975160586:84 , srv979013981:84 , srv968846524:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-04-09 04:01:43,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-09 04:01:43,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1006529008=0, srv950181901=96, srv649377819=86, srv752984093=89, srv1758164356=57, srv1184993849=13, srv1993918455=66, srv852243113=92, srv77444984=90, srv1908353965=63, srv1310883353=25, srv1661307226=53, srv428841657=78, srv1207725622=18, srv716168720=88, srv934612068=95, srv1224422342=20, srv156995844=42, srv1108413770=8, srv1727301666=54, srv2137645408=69, srv1441082292=34, srv57593493=85, srv1492924089=37, srv7790203=91, srv1506456826=38, srv1914676596=64, srv1204776373=17, srv499677418=80, srv1141722016=12, srv1332654499=27, srv1620915660=48, srv1033209364=2, srv1035065683=3, srv139474708=31, srv1557945466=40, srv2032656077=68, srv119118112=14, srv2012862228=67, srv1846941659=59, srv524379792=82, srv1197459545=16, srv1332382164=26, srv1256942895=22, srv56118254=84, srv1858844810=61, srv338808719=75, srv156810893=41, srv1830138711=58, srv1131899228=10, srv315759351=74, srv1284012166=23, srv894635753=94, srv1652913697=50, srv1751778448=56, srv1123895443=9, srv1290494592=24, srv975160586=98, srv1370558682=30, srv313212390=73, srv1598814734=45, srv1069301948=5, srv1088155190=6, srv232003577=72, srv1570484303=43, srv1654965362=51, srv1600815287=46, srv1342216643=28, srv1870938921=62, srv2139100000=70, srv1453680392=35, srv1486857011=36, srv219691665=71, srv1139537784=11, srv103515921=4, srv1580950724=44, srv1195293369=15, srv523692061=81, srv1644872171=49, srv968846524=97, srv1406940718=32, srv1608937401=47, srv884502374=93, srv1089634862=7, srv1407748336=33, srv1852123220=60, srv676554983=87, srv1234471143=21, srv365848177=76, srv1658991162=52, srv1970055917=65, srv461134156=79, srv1221154054=19, srv1554060389=39, srv1738384124=55, srv979013981=99, srv1356860474=29, srv372469652=77, srv541611378=83, srv1011754474=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-09 04:01:44,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-09 04:01:44,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-09 04:01:44,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-09 04:01:44,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-09 04:01:44,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-09 04:01:44,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-09 04:01:44,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-09 04:01:44,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-09 04:01:44,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-09 04:01:44,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-09 04:01:44,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-09 04:01:44,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-09 04:01:44,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-09 04:01:44,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-09 04:01:44,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-09 04:01:44,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-09 04:01:44,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-09 04:01:44,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-09 04:01:44,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-09 04:01:44,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-09 04:01:44,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-09 04:01:44,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-09 04:01:44,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-09 04:01:44,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-09 04:01:44,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-09 04:01:44,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-09 04:01:44,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-09 04:01:44,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-09 04:01:44,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-09 04:01:44,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-09 04:01:44,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-09 04:01:44,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-09 04:01:44,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-09 04:01:44,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-09 04:01:44,038 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-09 04:01:44,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-09 04:01:44,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-09 04:01:44,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-09 04:01:44,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-09 04:01:44,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-09 04:01:44,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-09 04:01:44,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-09 04:01:44,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-09 04:01:44,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-09 04:01:44,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-09 04:01:44,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-09 04:01:44,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-09 04:01:44,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-09 04:01:44,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-09 04:01:44,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-09 04:01:44,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-09 04:01:44,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-09 04:01:44,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-09 04:01:44,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-09 04:01:44,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-09 04:01:44,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-09 04:01:44,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-09 04:01:44,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-09 04:01:44,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-09 04:01:44,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-09 04:01:44,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-09 04:01:44,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-09 04:01:44,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-09 04:01:44,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-09 04:01:44,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-09 04:01:44,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-09 04:01:44,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-09 04:01:44,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-09 04:01:44,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-09 04:01:44,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-09 04:01:44,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-09 04:01:44,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-09 04:01:44,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-09 04:01:44,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-09 04:01:44,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-09 04:01:44,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-09 04:01:44,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-09 04:01:44,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-09 04:01:44,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-09 04:01:44,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-09 04:01:44,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-09 04:01:44,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-09 04:01:44,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-09 04:01:44,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-09 04:01:44,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-09 04:01:44,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-09 04:01:44,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-09 04:01:44,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-09 04:01:44,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-09 04:01:44,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-09 04:01:44,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-09 04:01:44,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-09 04:01:44,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-09 04:01:44,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-09 04:01:44,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-09 04:01:44,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-09 04:01:44,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-09 04:01:44,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-09 04:01:44,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-09 04:01:44,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-09 04:01:44,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-09 04:01:44,260 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-09 04:01:44,260 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-09 04:01:44,260 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-09 04:01:44,516 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 445 ms to try 5000 different iterations.  Found a solution that moves 4618 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18787614733869867. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041925533305320374, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05706042635568241, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5131111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041925533305320374, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18391666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24225, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-09 04:01:44,542 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1197459545:79 , srv1195293369:80 , srv156995844:80 , srv1088155190:81 , srv1554060389:81 , srv365848177:81 , srv499677418:81 , srv523692061:81 , srv1224422342:82 , srv1600815287:82 , srv1738384124:82 , srv676554983:82 , srv77444984:82 , srv950181901:82 , srv1035065683:83 , srv1221154054:83 , srv1557945466:83 , srv1751778448:83 , srv338808719:83 , srv852243113:83 , srv1089634862:84 , srv1332382164:84 , srv1342216643:84 , srv1370558682:84 , srv1453680392:84 , srv1620915660:84 , srv1870938921:84 , srv2139100000:84 , srv219691665:84 , srv524379792:84 , srv975160586:84 , srv1006529008:85 , srv1141722016:85 , srv1256942895:85 , srv1570484303:85 , srv1608937401:85 , srv1644872171:85 , srv1661307226:85 , srv1846941659:85 , srv894635753:85 , srv1011754474:86 , srv1123895443:86 , srv1139537784:86 , srv119118112:86 , srv1406940718:86 , srv1580950724:86 , srv1758164356:86 , srv1852123220:86 , srv2012862228:86 , srv2032656077:86 , srv461134156:86 , srv541611378:86 , srv7790203:86 , srv1234471143:87 , srv1441082292:87 , srv1486857011:87 , srv1652913697:87 , srv1727301666:87 , srv1830138711:87 , srv1858844810:87 , srv1914676596:87 , srv1993918455:87 , srv232003577:87 , srv428841657:87 , srv57593493:87 , srv752984093:87 , srv979013981:87 , srv1108413770:88 , srv1131899228:88 , srv1332654499:88 , srv139474708:88 , srv1908353965:88 , srv56118254:88 , srv649377819:88 , srv716168720:88 , srv884502374:88 , srv934612068:88 , srv1033209364:89 , srv1069301948:89 , srv1207725622:89 , srv1290494592:89 , srv1356860474:89 , srv1506456826:89 , srv1654965362:89 , srv315759351:89 , srv372469652:89 , srv103515921:90 , srv1284012166:90 , srv1310883353:90 , srv1407748336:90 , srv1492924089:90 , srv313212390:90 , srv1204776373:91 , srv2137645408:91 , srv1184993849:93 , srv1970055917:93 , srv156810893:95 , srv1658991162:96 , srv1598814734:99 , srv968846524:462 }
2023-04-09 04:01:44,548 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-09 04:01:44,549 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1006529008=0, srv950181901=96, srv649377819=86, srv752984093=89, srv1758164356=57, srv1184993849=13, srv1993918455=66, srv852243113=92, srv77444984=90, srv1908353965=63, srv1310883353=25, srv1661307226=53, srv428841657=78, srv1207725622=18, srv716168720=88, srv934612068=95, srv1224422342=20, srv156995844=42, srv1108413770=8, srv1727301666=54, srv2137645408=69, srv1441082292=34, srv57593493=85, srv1492924089=37, srv7790203=91, srv1506456826=38, srv1914676596=64, srv1204776373=17, srv499677418=80, srv1141722016=12, srv1332654499=27, srv1620915660=48, srv1033209364=2, srv1035065683=3, srv139474708=31, srv1557945466=40, srv2032656077=68, srv119118112=14, srv2012862228=67, srv1846941659=59, srv524379792=82, srv1197459545=16, srv1332382164=26, srv1256942895=22, srv56118254=84, srv1858844810=61, srv338808719=75, srv156810893=41, srv1830138711=58, srv1131899228=10, srv315759351=74, srv1284012166=23, srv894635753=94, srv1652913697=50, srv1751778448=56, srv1123895443=9, srv1290494592=24, srv975160586=98, srv1370558682=30, srv313212390=73, srv1598814734=45, srv1069301948=5, srv1088155190=6, srv232003577=72, srv1570484303=43, srv1654965362=51, srv1600815287=46, srv1342216643=28, srv1870938921=62, srv2139100000=70, srv1453680392=35, srv1486857011=36, srv219691665=71, srv1139537784=11, srv103515921=4, srv1580950724=44, srv1195293369=15, srv523692061=81, srv1644872171=49, srv968846524=97, srv1406940718=32, srv1608937401=47, srv884502374=93, srv1089634862=7, srv1407748336=33, srv1852123220=60, srv676554983=87, srv1234471143=21, srv365848177=76, srv1658991162=52, srv1970055917=65, srv461134156=79, srv1221154054=19, srv1554060389=39, srv1738384124=55, srv979013981=99, srv1356860474=29, srv372469652=77, srv541611378=83, srv1011754474=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-09 04:01:44,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-09 04:01:44,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-09 04:01:44,569 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-09 04:01:44,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-09 04:01:44,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-09 04:01:44,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-09 04:01:44,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-09 04:01:44,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-09 04:01:44,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-09 04:01:44,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-09 04:01:44,570 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-09 04:01:44,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-09 04:01:44,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-09 04:01:44,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-09 04:01:44,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-09 04:01:44,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-09 04:01:44,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-09 04:01:44,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-09 04:01:44,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-09 04:01:44,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-09 04:01:44,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-09 04:01:44,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-09 04:01:44,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-09 04:01:44,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-09 04:01:44,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-09 04:01:44,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-09 04:01:44,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-09 04:01:44,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-09 04:01:44,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-09 04:01:44,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-09 04:01:44,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-09 04:01:44,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-09 04:01:44,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-09 04:01:44,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-09 04:01:44,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-09 04:01:44,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-09 04:01:44,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-09 04:01:44,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-09 04:01:44,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-09 04:01:44,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-09 04:01:44,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-09 04:01:44,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-09 04:01:44,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-09 04:01:44,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-09 04:01:44,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-09 04:01:44,574 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-09 04:01:44,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-09 04:01:44,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-09 04:01:44,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-09 04:01:44,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-09 04:01:44,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-09 04:01:44,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-09 04:01:44,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-09 04:01:44,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-09 04:01:44,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-09 04:01:44,575 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-09 04:01:44,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-09 04:01:44,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-09 04:01:44,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-09 04:01:44,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-09 04:01:44,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-09 04:01:44,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-09 04:01:44,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-09 04:01:44,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-09 04:01:44,576 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-09 04:01:44,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-09 04:01:44,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-09 04:01:44,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-09 04:01:44,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-09 04:01:44,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-09 04:01:44,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-09 04:01:44,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-09 04:01:44,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-09 04:01:44,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-09 04:01:44,577 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-09 04:01:44,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-09 04:01:44,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-09 04:01:44,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-09 04:01:44,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-09 04:01:44,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-09 04:01:44,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-09 04:01:44,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-09 04:01:44,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-09 04:01:44,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-09 04:01:44,578 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-09 04:01:44,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-09 04:01:44,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-09 04:01:44,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-09 04:01:44,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-09 04:01:44,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-09 04:01:44,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-09 04:01:44,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-09 04:01:44,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-09 04:01:44,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-09 04:01:44,579 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-09 04:01:44,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-09 04:01:44,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-09 04:01:44,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-09 04:01:44,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-09 04:01:44,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-09 04:01:44,580 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-09 04:01:44,707 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041925533305320374, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05706042635568241, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041925533305320374, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18391666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24225, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-09 04:01:44,707 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-09 04:01:44,707 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18784381141156906, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.041925533305320374, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05706042635568241, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.041925533305320374, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.18391666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24225, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-09 04:01:44,931 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 327 ms to try 5000 different iterations.  Found a solution that moves 2636 regions; Going from a computed imbalance of 0.18784381141156906 to a new imbalance of 0.01619060807936481. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023852123720139534); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03885279710488263, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29288888888888887, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023852123720139534); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-09 04:01:44,940 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1598814734:73 , srv156810893:76 , srv1658991162:76 , srv1184993849:78 , srv103515921:80 , srv1356860474:80 , srv1310883353:81 , srv139474708:81 , srv1407748336:81 , srv1492924089:81 , srv1506456826:81 , srv1204776373:82 , srv1284012166:82 , srv1908353965:82 , srv1970055917:82 , srv1033209364:83 , srv1069301948:83 , srv1131899228:83 , srv1207725622:83 , srv1332654499:83 , srv232003577:83 , srv372469652:83 , srv716168720:83 , srv884502374:83 , srv1139537784:84 , srv1234471143:84 , srv1290494592:84 , srv1486857011:84 , srv1652913697:84 , srv1654965362:84 , srv1727301666:84 , srv1993918455:84 , srv315759351:84 , srv541611378:84 , srv56118254:84 , srv1406940718:85 , srv1570484303:85 , srv1914676596:85 , srv2137645408:85 , srv313212390:85 , srv428841657:85 , srv57593493:85 , srv894635753:85 , srv934612068:85 , srv1089634862:86 , srv1123895443:86 , srv1441082292:86 , srv1608937401:86 , srv1644872171:86 , srv1758164356:86 , srv1830138711:86 , srv1852123220:86 , srv1858844810:86 , srv2032656077:86 , srv524379792:86 , srv649377819:86 , srv7790203:86 , srv975160586:86 , srv979013981:86 , srv1035065683:87 , srv1141722016:87 , srv1332382164:87 , srv2012862228:87 , srv461134156:87 , srv1108413770:88 , srv1256942895:88 , srv1342216643:88 , srv1453680392:88 , srv1620915660:88 , srv1661307226:88 , srv1738384124:88 , srv1751778448:88 , srv1846941659:88 , srv1870938921:88 , srv2139100000:88 , srv338808719:88 , srv365848177:88 , srv752984093:88 , srv1006529008:89 , srv1011754474:89 , srv119118112:89 , srv1370558682:89 , srv1557945466:89 , srv1580950724:89 , srv1600815287:89 , srv219691665:89 , srv523692061:89 , srv77444984:89 , srv852243113:89 , srv950181901:89 , srv1221154054:90 , srv1224422342:90 , srv1554060389:90 , srv156995844:90 , srv1088155190:91 , srv1195293369:91 , srv676554983:91 , srv499677418:92 , srv1197459545:94 , srv968846524:524 }
2023-04-09 04:01:44,943 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-09 04:01:44,944 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1006529008=0, srv950181901=96, srv649377819=86, srv752984093=89, srv1758164356=57, srv1184993849=13, srv1993918455=66, srv852243113=92, srv77444984=90, srv1908353965=63, srv1310883353=25, srv1661307226=53, srv428841657=78, srv1207725622=18, srv716168720=88, srv934612068=95, srv1224422342=20, srv156995844=42, srv1108413770=8, srv1727301666=54, srv2137645408=69, srv1441082292=34, srv57593493=85, srv1492924089=37, srv7790203=91, srv1506456826=38, srv1914676596=64, srv1204776373=17, srv499677418=80, srv1141722016=12, srv1332654499=27, srv1620915660=48, srv1033209364=2, srv1035065683=3, srv139474708=31, srv1557945466=40, srv2032656077=68, srv119118112=14, srv2012862228=67, srv1846941659=59, srv524379792=82, srv1197459545=16, srv1332382164=26, srv1256942895=22, srv56118254=84, srv1858844810=61, srv338808719=75, srv156810893=41, srv1830138711=58, srv1131899228=10, srv315759351=74, srv1284012166=23, srv894635753=94, srv1652913697=50, srv1751778448=56, srv1123895443=9, srv1290494592=24, srv975160586=98, srv1370558682=30, srv313212390=73, srv1598814734=45, srv1069301948=5, srv1088155190=6, srv232003577=72, srv1570484303=43, srv1654965362=51, srv1600815287=46, srv1342216643=28, srv1870938921=62, srv2139100000=70, srv1453680392=35, srv1486857011=36, srv219691665=71, srv1139537784=11, srv103515921=4, srv1580950724=44, srv1195293369=15, srv523692061=81, srv1644872171=49, srv968846524=97, srv1406940718=32, srv1608937401=47, srv884502374=93, srv1089634862=7, srv1407748336=33, srv1852123220=60, srv676554983=87, srv1234471143=21, srv365848177=76, srv1658991162=52, srv1970055917=65, srv461134156=79, srv1221154054=19, srv1554060389=39, srv1738384124=55, srv979013981=99, srv1356860474=29, srv372469652=77, srv541611378=83, srv1011754474=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-09 04:01:44,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-09 04:01:44,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-09 04:01:44,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-09 04:01:44,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-09 04:01:44,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-09 04:01:44,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-09 04:01:44,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-09 04:01:44,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-09 04:01:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-09 04:01:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-09 04:01:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-09 04:01:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-09 04:01:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-09 04:01:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-09 04:01:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-09 04:01:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-09 04:01:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-09 04:01:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-09 04:01:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-09 04:01:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-09 04:01:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-09 04:01:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-09 04:01:44,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-09 04:01:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-09 04:01:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-09 04:01:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-09 04:01:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-09 04:01:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-09 04:01:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-09 04:01:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-09 04:01:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-09 04:01:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-09 04:01:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-09 04:01:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-09 04:01:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-09 04:01:44,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-09 04:01:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-09 04:01:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-09 04:01:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-09 04:01:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-09 04:01:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-09 04:01:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-09 04:01:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-09 04:01:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-09 04:01:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-09 04:01:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-09 04:01:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-09 04:01:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-09 04:01:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-09 04:01:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-09 04:01:44,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-09 04:01:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-09 04:01:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-09 04:01:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-09 04:01:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-09 04:01:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-09 04:01:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-09 04:01:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-09 04:01:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-09 04:01:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-09 04:01:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-09 04:01:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-09 04:01:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-09 04:01:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-09 04:01:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-09 04:01:44,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-09 04:01:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-09 04:01:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-09 04:01:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-09 04:01:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-09 04:01:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-09 04:01:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-09 04:01:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-09 04:01:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-09 04:01:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-09 04:01:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-09 04:01:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-09 04:01:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-09 04:01:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-09 04:01:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-09 04:01:44,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-09 04:01:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-09 04:01:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-09 04:01:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-09 04:01:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-09 04:01:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-09 04:01:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-09 04:01:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-09 04:01:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-09 04:01:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-09 04:01:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-09 04:01:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-09 04:01:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-09 04:01:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-09 04:01:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-09 04:01:44,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-09 04:01:44,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-09 04:01:44,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-09 04:01:44,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-09 04:01:44,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-09 04:01:44,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-09 04:01:45,058 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023852123720139534); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03885279710488263, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023852123720139534); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-09 04:01:45,058 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-09 04:01:45,058 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.016172150412861193, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023852123720139534); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03885279710488263, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023852123720139534); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.013083333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04558333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-09 04:01:45,218 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 237 ms to try 5000 different iterations.  Found a solution that moves 1030 regions; Going from a computed imbalance of 0.016172150412861193 to a new imbalance of 9.623318575190698E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008456073273622864); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011468984183626066); MoveCostFunction : (multiplier=7.0, imbalance=0.11444444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008456073273622864); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-09 04:01:45,220 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1970055917:81 , srv752984093:81 , srv1108413770:82 , srv1580950724:82 , srv2137645408:82 , srv313212390:82 , srv1006529008:83 , srv1011754474:83 , srv119118112:83 , srv1221154054:83 , srv1033209364:84 , srv1197459545:84 , srv1204776373:84 , srv1256942895:84 , srv1290494592:84 , srv1342216643:84 , srv1370558682:84 , srv1441082292:84 , srv1654965362:84 , srv1661307226:84 , srv1852123220:84 , srv1858844810:84 , srv219691665:84 , srv315759351:84 , srv461134156:84 , srv499677418:84 , srv649377819:84 , srv934612068:84 , srv1069301948:85 , srv1088155190:85 , srv1123895443:85 , srv1207725622:85 , srv1453680392:85 , srv1598814734:85 , srv1620915660:85 , srv1652913697:85 , srv1758164356:85 , srv1830138711:85 , srv1846941659:85 , srv1870938921:85 , srv1914676596:85 , srv2012862228:85 , srv2032656077:85 , srv2139100000:85 , srv372469652:85 , srv428841657:85 , srv56118254:85 , srv676554983:85 , srv979013981:85 , srv103515921:86 , srv1131899228:86 , srv1139537784:86 , srv1141722016:86 , srv1195293369:86 , srv1224422342:86 , srv1284012166:86 , srv1310883353:86 , srv1332654499:86 , srv1406940718:86 , srv1407748336:86 , srv1554060389:86 , srv1557945466:86 , srv156810893:86 , srv1570484303:86 , srv1600815287:86 , srv1644872171:86 , srv1658991162:86 , srv1751778448:86 , srv523692061:86 , srv524379792:86 , srv77444984:86 , srv7790203:86 , srv852243113:86 , srv884502374:86 , srv950181901:86 , srv1089634862:87 , srv1184993849:87 , srv1234471143:87 , srv1332382164:87 , srv1486857011:87 , srv1492924089:87 , srv156995844:87 , srv1608937401:87 , srv1908353965:87 , srv232003577:87 , srv338808719:87 , srv365848177:87 , srv57593493:87 , srv716168720:87 , srv894635753:87 , srv975160586:87 , srv1035065683:88 , srv1356860474:88 , srv1506456826:88 , srv1727301666:88 , srv1738384124:88 , srv1993918455:88 , srv139474708:89 , srv541611378:89 , srv968846524:547 }
2023-04-09 04:01:45,224 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-09 04:01:45,225 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1006529008=0, srv950181901=96, srv649377819=86, srv752984093=89, srv1758164356=57, srv1184993849=13, srv1993918455=66, srv852243113=92, srv77444984=90, srv1908353965=63, srv1310883353=25, srv1661307226=53, srv428841657=78, srv1207725622=18, srv716168720=88, srv934612068=95, srv1224422342=20, srv156995844=42, srv1108413770=8, srv1727301666=54, srv2137645408=69, srv1441082292=34, srv57593493=85, srv1492924089=37, srv7790203=91, srv1506456826=38, srv1914676596=64, srv1204776373=17, srv499677418=80, srv1141722016=12, srv1332654499=27, srv1620915660=48, srv1033209364=2, srv1035065683=3, srv139474708=31, srv1557945466=40, srv2032656077=68, srv119118112=14, srv2012862228=67, srv1846941659=59, srv524379792=82, srv1197459545=16, srv1332382164=26, srv1256942895=22, srv56118254=84, srv1858844810=61, srv338808719=75, srv156810893=41, srv1830138711=58, srv1131899228=10, srv315759351=74, srv1284012166=23, srv894635753=94, srv1652913697=50, srv1751778448=56, srv1123895443=9, srv1290494592=24, srv975160586=98, srv1370558682=30, srv313212390=73, srv1598814734=45, srv1069301948=5, srv1088155190=6, srv232003577=72, srv1570484303=43, srv1654965362=51, srv1600815287=46, srv1342216643=28, srv1870938921=62, srv2139100000=70, srv1453680392=35, srv1486857011=36, srv219691665=71, srv1139537784=11, srv103515921=4, srv1580950724=44, srv1195293369=15, srv523692061=81, srv1644872171=49, srv968846524=97, srv1406940718=32, srv1608937401=47, srv884502374=93, srv1089634862=7, srv1407748336=33, srv1852123220=60, srv676554983=87, srv1234471143=21, srv365848177=76, srv1658991162=52, srv1970055917=65, srv461134156=79, srv1221154054=19, srv1554060389=39, srv1738384124=55, srv979013981=99, srv1356860474=29, srv372469652=77, srv541611378=83, srv1011754474=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-09 04:01:45,237 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-09 04:01:45,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-09 04:01:45,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-09 04:01:45,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-09 04:01:45,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-09 04:01:45,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-09 04:01:45,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-09 04:01:45,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-09 04:01:45,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-09 04:01:45,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-09 04:01:45,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-09 04:01:45,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-09 04:01:45,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-09 04:01:45,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-09 04:01:45,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-09 04:01:45,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-09 04:01:45,238 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-09 04:01:45,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-09 04:01:45,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-09 04:01:45,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-09 04:01:45,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-09 04:01:45,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-09 04:01:45,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-09 04:01:45,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-09 04:01:45,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-09 04:01:45,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-09 04:01:45,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-09 04:01:45,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-09 04:01:45,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-09 04:01:45,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-09 04:01:45,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-09 04:01:45,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-09 04:01:45,239 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-09 04:01:45,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-09 04:01:45,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-09 04:01:45,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-09 04:01:45,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-09 04:01:45,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-09 04:01:45,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-09 04:01:45,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-09 04:01:45,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-09 04:01:45,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-09 04:01:45,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-09 04:01:45,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-09 04:01:45,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-09 04:01:45,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-09 04:01:45,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-09 04:01:45,240 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-09 04:01:45,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-09 04:01:45,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-09 04:01:45,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-09 04:01:45,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-09 04:01:45,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-09 04:01:45,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-09 04:01:45,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-09 04:01:45,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-09 04:01:45,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-09 04:01:45,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-09 04:01:45,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-09 04:01:45,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-09 04:01:45,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-09 04:01:45,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-09 04:01:45,241 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-09 04:01:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-09 04:01:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-09 04:01:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-09 04:01:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-09 04:01:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-09 04:01:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-09 04:01:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-09 04:01:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-09 04:01:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-09 04:01:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-09 04:01:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-09 04:01:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-09 04:01:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-09 04:01:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-09 04:01:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-09 04:01:45,242 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-09 04:01:45,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-09 04:01:45,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-09 04:01:45,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-09 04:01:45,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-09 04:01:45,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-09 04:01:45,322 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008456073273622864); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011468984183626066); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008456073273622864); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-09 04:01:45,322 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-09 04:01:45,323 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.55119643369326E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008456073273622864); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.011468984183626066); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008456073273622864); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=4.166666666666667E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-09 04:01:45,539 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 287 ms to try 5000 different iterations.  Found a solution that moves 336 regions; Going from a computed imbalance of 9.55119643369326E-4 to a new imbalance of 4.533238268046318E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.037333333333333336, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-09 04:01:45,540 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv57593493:82 , srv1011754474:83 , srv1993918455:83 , srv541611378:83 , srv1035065683:84 , srv1108413770:84 , srv1123895443:84 , srv1141722016:84 , srv1184993849:84 , srv1204776373:84 , srv1234471143:84 , srv1284012166:84 , srv1332382164:84 , srv1356860474:84 , srv139474708:84 , srv1486857011:84 , srv1492924089:84 , srv1557945466:84 , srv1608937401:84 , srv1644872171:84 , srv1658991162:84 , srv1727301666:84 , srv1738384124:84 , srv1830138711:84 , srv1914676596:84 , srv2012862228:84 , srv338808719:84 , srv649377819:84 , srv676554983:84 , srv716168720:84 , srv77444984:84 , srv7790203:84 , srv852243113:84 , srv884502374:84 , srv934612068:84 , srv975160586:84 , srv979013981:84 , srv1069301948:85 , srv1088155190:85 , srv1089634862:85 , srv1131899228:85 , srv1139537784:85 , srv119118112:85 , srv1195293369:85 , srv1197459545:85 , srv1224422342:85 , srv1256942895:85 , srv1290494592:85 , srv1310883353:85 , srv1332654499:85 , srv1406940718:85 , srv1407748336:85 , srv1441082292:85 , srv1453680392:85 , srv1506456826:85 , srv1554060389:85 , srv156810893:85 , srv156995844:85 , srv1598814734:85 , srv1600815287:85 , srv1620915660:85 , srv1654965362:85 , srv1661307226:85 , srv1751778448:85 , srv1846941659:85 , srv1858844810:85 , srv1870938921:85 , srv1908353965:85 , srv2032656077:85 , srv2137645408:85 , srv2139100000:85 , srv219691665:85 , srv232003577:85 , srv313212390:85 , srv315759351:85 , srv372469652:85 , srv428841657:85 , srv461134156:85 , srv499677418:85 , srv524379792:85 , srv56118254:85 , srv894635753:85 , srv950181901:85 , srv1006529008:86 , srv1033209364:86 , srv103515921:86 , srv1207725622:86 , srv1342216643:86 , srv1370558682:86 , srv1570484303:86 , srv1580950724:86 , srv1652913697:86 , srv1758164356:86 , srv1852123220:86 , srv1970055917:86 , srv365848177:86 , srv523692061:86 , srv752984093:86 , srv1221154054:87 , srv968846524:610 }
2023-04-09 04:01:45,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-09 04:01:45,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1006529008=0, srv950181901=96, srv649377819=86, srv752984093=89, srv1758164356=57, srv1184993849=13, srv1993918455=66, srv852243113=92, srv77444984=90, srv1908353965=63, srv1310883353=25, srv1661307226=53, srv428841657=78, srv1207725622=18, srv716168720=88, srv934612068=95, srv1224422342=20, srv156995844=42, srv1108413770=8, srv1727301666=54, srv2137645408=69, srv1441082292=34, srv57593493=85, srv1492924089=37, srv7790203=91, srv1506456826=38, srv1914676596=64, srv1204776373=17, srv499677418=80, srv1141722016=12, srv1332654499=27, srv1620915660=48, srv1033209364=2, srv1035065683=3, srv139474708=31, srv1557945466=40, srv2032656077=68, srv119118112=14, srv2012862228=67, srv1846941659=59, srv524379792=82, srv1197459545=16, srv1332382164=26, srv1256942895=22, srv56118254=84, srv1858844810=61, srv338808719=75, srv156810893=41, srv1830138711=58, srv1131899228=10, srv315759351=74, srv1284012166=23, srv894635753=94, srv1652913697=50, srv1751778448=56, srv1123895443=9, srv1290494592=24, srv975160586=98, srv1370558682=30, srv313212390=73, srv1598814734=45, srv1069301948=5, srv1088155190=6, srv232003577=72, srv1570484303=43, srv1654965362=51, srv1600815287=46, srv1342216643=28, srv1870938921=62, srv2139100000=70, srv1453680392=35, srv1486857011=36, srv219691665=71, srv1139537784=11, srv103515921=4, srv1580950724=44, srv1195293369=15, srv523692061=81, srv1644872171=49, srv968846524=97, srv1406940718=32, srv1608937401=47, srv884502374=93, srv1089634862=7, srv1407748336=33, srv1852123220=60, srv676554983=87, srv1234471143=21, srv365848177=76, srv1658991162=52, srv1970055917=65, srv461134156=79, srv1221154054=19, srv1554060389=39, srv1738384124=55, srv979013981=99, srv1356860474=29, srv372469652=77, srv541611378=83, srv1011754474=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-09 04:01:45,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-09 04:01:45,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-09 04:01:45,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-09 04:01:45,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-09 04:01:45,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-09 04:01:45,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-09 04:01:45,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-09 04:01:45,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-09 04:01:45,552 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-09 04:01:45,553 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-09 04:01:45,554 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-09 04:01:45,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-09 04:01:45,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-09 04:01:45,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-09 04:01:45,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-09 04:01:45,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-09 04:01:45,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-09 04:01:45,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-09 04:01:45,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-09 04:01:45,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-09 04:01:45,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-09 04:01:45,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-09 04:01:45,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-09 04:01:45,555 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-09 04:01:45,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-09 04:01:45,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-09 04:01:45,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-09 04:01:45,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-09 04:01:45,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-09 04:01:45,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-09 04:01:45,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-09 04:01:45,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-09 04:01:45,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-09 04:01:45,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-09 04:01:45,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-09 04:01:45,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-09 04:01:45,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-09 04:01:45,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-09 04:01:45,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-09 04:01:45,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-09 04:01:45,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-09 04:01:45,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-09 04:01:45,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-09 04:01:45,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-09 04:01:45,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-09 04:01:45,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-09 04:01:45,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-09 04:01:45,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-09 04:01:45,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-09 04:01:45,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-09 04:01:45,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-09 04:01:45,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-09 04:01:45,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-09 04:01:45,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-09 04:01:45,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-09 04:01:45,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-09 04:01:45,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-09 04:01:45,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-09 04:01:45,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-09 04:01:45,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-09 04:01:45,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-09 04:01:45,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-09 04:01:45,559 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-09 04:01:45,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-09 04:01:45,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-09 04:01:45,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-09 04:01:45,560 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-09 04:01:45,637 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-09 04:01:45,637 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-04-09 04:01:45,637 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.2979660394721455E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.694119368783758E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022721660362024225); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.694119368783758E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=3.333333333333333E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-04-09 04:01:45,777 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 209 ms to try 5000 different iterations.  Found a solution that moves 48 regions; Going from a computed imbalance of 4.2979660394721455E-5 to a new imbalance of 7.080172237509337E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.005333333333333333); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-09 04:01:45,778 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1006529008:83 , srv119118112:83 , srv1207725622:83 , srv1221154054:83 , srv1224422342:83 , srv1370558682:83 , srv1506456826:83 , srv1580950724:83 , srv1727301666:83 , srv1758164356:83 , srv1846941659:83 , srv2137645408:83 , srv1033209364:84 , srv1035065683:84 , srv103515921:84 , srv1069301948:84 , srv1088155190:84 , srv1089634862:84 , srv1108413770:84 , srv1131899228:84 , srv1141722016:84 , srv1184993849:84 , srv1195293369:84 , srv1197459545:84 , srv1234471143:84 , srv1256942895:84 , srv1284012166:84 , srv1290494592:84 , srv1310883353:84 , srv1332382164:84 , srv1332654499:84 , srv1342216643:84 , srv139474708:84 , srv1406940718:84 , srv1407748336:84 , srv1441082292:84 , srv1453680392:84 , srv1486857011:84 , srv1492924089:84 , srv1554060389:84 , srv1557945466:84 , srv156810893:84 , srv156995844:84 , srv1570484303:84 , srv1598814734:84 , srv1600815287:84 , srv1608937401:84 , srv1620915660:84 , srv1652913697:84 , srv1654965362:84 , srv1658991162:84 , srv1661307226:84 , srv1738384124:84 , srv1751778448:84 , srv1830138711:84 , srv1852123220:84 , srv1858844810:84 , srv1870938921:84 , srv1908353965:84 , srv1970055917:84 , srv1993918455:84 , srv2012862228:84 , srv2032656077:84 , srv2139100000:84 , srv219691665:84 , srv232003577:84 , srv313212390:84 , srv315759351:84 , srv338808719:84 , srv365848177:84 , srv372469652:84 , srv428841657:84 , srv461134156:84 , srv499677418:84 , srv523692061:84 , srv541611378:84 , srv56118254:84 , srv649377819:84 , srv676554983:84 , srv716168720:84 , srv752984093:84 , srv7790203:84 , srv852243113:84 , srv894635753:84 , srv950181901:84 , srv979013981:84 , srv1011754474:85 , srv1123895443:85 , srv1139537784:85 , srv1204776373:85 , srv1356860474:85 , srv1644872171:85 , srv1914676596:85 , srv524379792:85 , srv57593493:85 , srv77444984:85 , srv884502374:85 , srv934612068:85 , srv975160586:85 , srv968846524:683 }
2023-04-09 04:01:45,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-09 04:01:45,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1006529008=0, srv950181901=96, srv649377819=86, srv752984093=89, srv1758164356=57, srv1184993849=13, srv1993918455=66, srv852243113=92, srv77444984=90, srv1908353965=63, srv1310883353=25, srv1661307226=53, srv428841657=78, srv1207725622=18, srv716168720=88, srv934612068=95, srv1224422342=20, srv156995844=42, srv1108413770=8, srv1727301666=54, srv2137645408=69, srv1441082292=34, srv57593493=85, srv1492924089=37, srv7790203=91, srv1506456826=38, srv1914676596=64, srv1204776373=17, srv499677418=80, srv1141722016=12, srv1332654499=27, srv1620915660=48, srv1033209364=2, srv1035065683=3, srv139474708=31, srv1557945466=40, srv2032656077=68, srv119118112=14, srv2012862228=67, srv1846941659=59, srv524379792=82, srv1197459545=16, srv1332382164=26, srv1256942895=22, srv56118254=84, srv1858844810=61, srv338808719=75, srv156810893=41, srv1830138711=58, srv1131899228=10, srv315759351=74, srv1284012166=23, srv894635753=94, srv1652913697=50, srv1751778448=56, srv1123895443=9, srv1290494592=24, srv975160586=98, srv1370558682=30, srv313212390=73, srv1598814734=45, srv1069301948=5, srv1088155190=6, srv232003577=72, srv1570484303=43, srv1654965362=51, srv1600815287=46, srv1342216643=28, srv1870938921=62, srv2139100000=70, srv1453680392=35, srv1486857011=36, srv219691665=71, srv1139537784=11, srv103515921=4, srv1580950724=44, srv1195293369=15, srv523692061=81, srv1644872171=49, srv968846524=97, srv1406940718=32, srv1608937401=47, srv884502374=93, srv1089634862=7, srv1407748336=33, srv1852123220=60, srv676554983=87, srv1234471143=21, srv365848177=76, srv1658991162=52, srv1970055917=65, srv461134156=79, srv1221154054=19, srv1554060389=39, srv1738384124=55, srv979013981=99, srv1356860474=29, srv372469652=77, srv541611378=83, srv1011754474=1} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-09 04:01:45,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-09 04:01:45,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-04-09 04:01:45,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-04-09 04:01:45,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-04-09 04:01:45,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-04-09 04:01:45,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-04-09 04:01:45,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-04-09 04:01:45,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-04-09 04:01:45,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-04-09 04:01:45,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-04-09 04:01:45,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-04-09 04:01:45,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-04-09 04:01:45,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-04-09 04:01:45,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-04-09 04:01:45,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-04-09 04:01:45,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-04-09 04:01:45,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-04-09 04:01:45,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-04-09 04:01:45,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-04-09 04:01:45,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-04-09 04:01:45,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-04-09 04:01:45,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-04-09 04:01:45,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-04-09 04:01:45,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-04-09 04:01:45,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-04-09 04:01:45,874 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-09 04:01:45,875 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1006529008:83 , srv119118112:83 , srv1207725622:83 , srv1221154054:83 , srv1224422342:83 , srv1370558682:83 , srv1506456826:83 , srv1580950724:83 , srv1727301666:83 , srv1758164356:83 , srv1846941659:83 , srv2137645408:83 , srv1033209364:84 , srv1035065683:84 , srv103515921:84 , srv1069301948:84 , srv1088155190:84 , srv1089634862:84 , srv1108413770:84 , srv1131899228:84 , srv1141722016:84 , srv1184993849:84 , srv1195293369:84 , srv1197459545:84 , srv1234471143:84 , srv1256942895:84 , srv1284012166:84 , srv1290494592:84 , srv1310883353:84 , srv1332382164:84 , srv1332654499:84 , srv1342216643:84 , srv139474708:84 , srv1406940718:84 , srv1407748336:84 , srv1441082292:84 , srv1453680392:84 , srv1486857011:84 , srv1492924089:84 , srv1554060389:84 , srv1557945466:84 , srv156810893:84 , srv156995844:84 , srv1570484303:84 , srv1598814734:84 , srv1600815287:84 , srv1608937401:84 , srv1620915660:84 , srv1652913697:84 , srv1654965362:84 , srv1658991162:84 , srv1661307226:84 , srv1738384124:84 , srv1751778448:84 , srv1830138711:84 , srv1852123220:84 , srv1858844810:84 , srv1870938921:84 , srv1908353965:84 , srv1970055917:84 , srv1993918455:84 , srv2012862228:84 , srv2032656077:84 , srv2139100000:84 , srv219691665:84 , srv232003577:84 , srv313212390:84 , srv315759351:84 , srv338808719:84 , srv365848177:84 , srv372469652:84 , srv428841657:84 , srv461134156:84 , srv499677418:84 , srv523692061:84 , srv541611378:84 , srv56118254:84 , srv649377819:84 , srv676554983:84 , srv716168720:84 , srv752984093:84 , srv7790203:84 , srv852243113:84 , srv894635753:84 , srv950181901:84 , srv979013981:84 , srv1011754474:85 , srv1123895443:85 , srv1139537784:85 , srv1204776373:85 , srv1356860474:85 , srv1644872171:85 , srv1914676596:85 , srv524379792:85 , srv57593493:85 , srv77444984:85 , srv884502374:85 , srv934612068:85 , srv975160586:85 , srv968846524:683 }
2023-04-09 04:01:45,896 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=163 (was 163), ProcessCount=167 (was 167), AvailableMemoryMB=4193 (was 4687)
2023-04-09 04:01:45,904 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=163, ProcessCount=167, AvailableMemoryMB=4192
2023-04-09 04:01:45,905 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-04-09 04:01:45,905 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-04-09 04:01:45,915 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1006737790:84 , srv1027887939:84 , srv1115908144:84 , srv1137542573:84 , srv1250417372:84 , srv1480200081:84 , srv1562108941:84 , srv1591980999:84 , srv1603057467:84 , srv1603864360:84 , srv1703851864:84 , srv1807289228:84 , srv1840891526:84 , srv2027096095:84 , srv2037699331:84 , srv2111872466:84 , srv2128476872:84 , srv2139617387:84 , srv278226041:84 , srv371653822:84 , srv415851386:84 , srv458538094:84 , srv470544382:84 , srv506295897:84 , srv590824335:84 , srv623520075:84 , srv796272287:84 , srv805928320:84 , srv968476507:84 , srv899490209:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-04-09 04:01:45,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-09 04:01:45,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1603864360=9, srv805928320=27, srv899490209=28, srv1591980999=7, srv458538094=21, srv1703851864=10, srv1807289228=11, srv2037699331=14, srv623520075=25, srv1027887939=1, srv2139617387=17, srv796272287=26, srv1480200081=5, srv1840891526=12, srv470544382=22, srv968476507=29, srv1562108941=6, srv1115908144=2, srv2128476872=16, srv1603057467=8, srv371653822=19, srv415851386=20, srv506295897=23, srv590824335=24, srv278226041=18, srv1137542573=3, srv1250417372=4, srv2111872466=15, srv2027096095=13, srv1006737790=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-09 04:01:45,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-09 04:01:45,929 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-09 04:01:45,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-09 04:01:45,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-09 04:01:45,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-09 04:01:45,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-09 04:01:45,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-09 04:01:45,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-09 04:01:45,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-09 04:01:45,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-09 04:01:45,930 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-09 04:01:45,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-09 04:01:45,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-09 04:01:45,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-09 04:01:45,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-09 04:01:45,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-09 04:01:45,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-09 04:01:45,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-09 04:01:45,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-09 04:01:45,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-09 04:01:45,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-09 04:01:45,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-09 04:01:45,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-09 04:01:45,931 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-09 04:01:45,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-09 04:01:45,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-09 04:01:45,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-09 04:01:45,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-09 04:01:45,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-09 04:01:45,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-09 04:01:45,932 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-09 04:01:45,953 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1465225686545515, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-09 04:01:45,954 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912654731093707, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1465225686545515, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-04-09 04:03:45,936 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9277373 different iterations.  Found a solution that moves 1835 regions; Going from a computed imbalance of 0.9912654731093707 to a new imbalance of 4.2829815420000605E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6796296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-09 04:03:45,944 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1006737790:90 , srv1027887939:90 , srv1115908144:90 , srv1137542573:90 , srv1250417372:90 , srv1480200081:90 , srv1562108941:90 , srv1591980999:90 , srv1603057467:90 , srv1603864360:90 , srv1703851864:90 , srv1807289228:90 , srv1840891526:90 , srv2027096095:90 , srv2037699331:90 , srv2111872466:90 , srv2128476872:90 , srv2139617387:90 , srv278226041:90 , srv371653822:90 , srv415851386:90 , srv458538094:90 , srv470544382:90 , srv506295897:90 , srv590824335:90 , srv623520075:90 , srv796272287:90 , srv805928320:90 , srv899490209:90 , srv968476507:90 }
2023-04-09 04:03:45,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-04-09 04:03:45,946 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1603864360=9, srv805928320=27, srv899490209=28, srv1591980999=7, srv458538094=21, srv1703851864=10, srv1807289228=11, srv2037699331=14, srv623520075=25, srv1027887939=1, srv2139617387=17, srv796272287=26, srv1480200081=5, srv1840891526=12, srv470544382=22, srv968476507=29, srv1562108941=6, srv1115908144=2, srv2128476872=16, srv1603057467=8, srv371653822=19, srv415851386=20, srv506295897=23, srv590824335=24, srv278226041=18, srv1137542573=3, srv1250417372=4, srv2111872466=15, srv2027096095=13, srv1006737790=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-04-09 04:03:45,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-04-09 04:03:45,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-04-09 04:03:45,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-04-09 04:03:45,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-04-09 04:03:45,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-04-09 04:03:45,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-04-09 04:03:45,951 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-04-09 04:03:45,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-04-09 04:03:45,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-04-09 04:03:45,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-04-09 04:03:45,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-04-09 04:03:45,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-04-09 04:03:45,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-04-09 04:03:45,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-04-09 04:03:45,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-04-09 04:03:45,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-04-09 04:03:45,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-04-09 04:03:45,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-04-09 04:03:45,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-04-09 04:03:45,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-04-09 04:03:45,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-04-09 04:03:45,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-04-09 04:03:45,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-04-09 04:03:45,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-04-09 04:03:45,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-04-09 04:03:45,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-04-09 04:03:45,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-04-09 04:03:45,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-04-09 04:03:45,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-04-09 04:03:45,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-04-09 04:03:45,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-04-09 04:03:45,969 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-04-09 04:03:45,969 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1006737790:90 , srv1027887939:90 , srv1115908144:90 , srv1137542573:90 , srv1250417372:90 , srv1480200081:90 , srv1562108941:90 , srv1591980999:90 , srv1603057467:90 , srv1603864360:90 , srv1703851864:90 , srv1807289228:90 , srv1840891526:90 , srv2027096095:90 , srv2037699331:90 , srv2111872466:90 , srv2128476872:90 , srv2139617387:90 , srv278226041:90 , srv371653822:90 , srv415851386:90 , srv458538094:90 , srv470544382:90 , srv506295897:90 , srv590824335:90 , srv623520075:90 , srv796272287:90 , srv805928320:90 , srv899490209:90 , srv968476507:90 }
2023-04-09 04:03:45,985 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=204 (was 163) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 167) - ProcessCount LEAK? -, AvailableMemoryMB=4196 (was 4192) - AvailableMemoryMB LEAK? -