2023-09-14 23:53:12,717 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-09-14 23:53:13,026 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-09-14 23:53:13,136 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-09-14 23:53:13,205 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-14 23:53:13,212 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-14 23:53:13,225 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=337, ProcessCount=169, AvailableMemoryMB=10267
2023-09-14 23:53:13,227 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-14 23:53:13,228 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-09-14 23:53:13,396 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1010994268:84 , srv1027846200:84 , srv1031321149:84 , srv1033366048:84 , srv1062248711:84 , srv1082824981:84 , srv1147607115:84 , srv1180094587:84 , srv1188619468:84 , srv1207042735:84 , srv1263581844:84 , srv1370852155:84 , srv1393746149:84 , srv1417116795:84 , srv1423536980:84 , srv1430988569:84 , srv1432405231:84 , srv1436882930:84 , srv1451429348:84 , srv1458447263:84 , srv1466755558:84 , srv150026344:84 , srv1505827474:84 , srv1520384978:84 , srv1526379784:84 , srv1551447105:84 , srv1553395038:84 , srv1584577667:84 , srv1593500481:84 , srv1605385265:84 , srv1624585469:84 , srv1644514596:84 , srv1660657710:84 , srv1664651676:84 , srv1678447537:84 , srv1690659430:84 , srv1703867815:84 , srv1712785215:84 , srv177220054:84 , srv1794856270:84 , srv1813513855:84 , srv18374836:84 , srv1869189048:84 , srv1901633749:84 , srv1937062381:84 , srv1938783145:84 , srv1980849281:84 , srv1982631872:84 , srv202530211:84 , srv2041105836:84 , srv2046686948:84 , srv2056214813:84 , srv2065558605:84 , srv2080711270:84 , srv2127561277:84 , srv238557422:84 , srv309801858:84 , srv321385986:84 , srv339772613:84 , srv343503108:84 , srv36446101:84 , srv390841821:84 , srv404956531:84 , srv408801329:84 , srv419468499:84 , srv473917378:84 , srv500455395:84 , srv519165520:84 , srv554698718:84 , srv554854346:84 , srv588453639:84 , srv593784087:84 , srv594664804:84 , srv602912654:84 , srv614049239:84 , srv61897985:84 , srv630692188:84 , srv638407791:84 , srv658801312:84 , srv680301615:84 , srv688372241:84 , srv696092676:84 , srv70315645:84 , srv715767378:84 , srv737162939:84 , srv740928734:84 , srv756331166:84 , srv758616378:84 , srv783173800:84 , srv783802171:84 , srv792437889:84 , srv793568030:84 , srv818635884:84 , srv822540784:84 , srv858118781:84 , srv866816453:84 , srv910172570:84 , srv965055491:84 , srv975276356:84 , srv848322282:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-09-14 23:53:13,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-14 23:53:13,413 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv858118781=95, srv1180094587=7, srv758616378=87, srv658801312=78, srv70315645=82, srv61897985=75, srv1451429348=18, srv822540784=93, srv756331166=86, srv1458447263=19, srv1432405231=16, srv2127561277=54, srv1010994268=0, srv1436882930=17, srv737162939=84, srv593784087=71, srv1938783145=45, srv783173800=88, srv1551447105=25, srv630692188=76, srv588453639=70, srv404956531=62, srv1393746149=12, srv2065558605=52, srv1937062381=44, srv1370852155=11, srv1703867815=36, srv910172570=97, srv177220054=38, srv408801329=63, srv1553395038=26, srv554698718=68, srv1644514596=31, srv1033366048=3, srv150026344=21, srv1605385265=29, srv975276356=99, srv1660657710=32, srv2056214813=51, srv390841821=61, srv339772613=58, srv419468499=64, srv1147607115=6, srv1690659430=35, srv473917378=65, srv1466755558=20, srv519165520=67, srv309801858=56, srv1980849281=46, srv715767378=83, srv793568030=91, srv594664804=72, srv36446101=60, srv500455395=66, srv1263581844=10, srv638407791=77, srv1869189048=42, srv1712785215=37, srv2080711270=53, srv554854346=69, srv1031321149=2, srv1982631872=47, srv1027846200=1, srv2041105836=49, srv18374836=41, srv602912654=73, srv783802171=89, srv1188619468=8, srv343503108=59, srv1062248711=4, srv238557422=55, srv202530211=48, srv321385986=57, srv1593500481=28, srv1901633749=43, srv1430988569=15, srv965055491=98, srv848322282=94, srv1207042735=9, srv1520384978=23, srv1526379784=24, srv866816453=96, srv1794856270=39, srv1664651676=33, srv1624585469=30, srv818635884=92, srv680301615=79, srv614049239=74, srv1584577667=27, srv792437889=90, srv1417116795=13, srv696092676=81, srv1082824981=5, srv1678447537=34, srv740928734=85, srv2046686948=50, srv688372241=80, srv1813513855=40, srv1423536980=14, srv1505827474=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-14 23:53:13,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-14 23:53:13,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-14 23:53:13,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-14 23:53:13,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-14 23:53:13,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-14 23:53:13,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-14 23:53:13,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-14 23:53:13,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-14 23:53:13,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-14 23:53:13,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-14 23:53:13,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-14 23:53:13,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-14 23:53:13,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-14 23:53:13,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-14 23:53:13,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-14 23:53:13,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-14 23:53:13,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-14 23:53:13,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-14 23:53:13,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-14 23:53:13,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-14 23:53:13,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-14 23:53:13,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-14 23:53:13,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-14 23:53:13,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-14 23:53:13,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-14 23:53:13,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-14 23:53:13,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-14 23:53:13,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-14 23:53:13,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-14 23:53:13,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-14 23:53:13,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-14 23:53:13,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-14 23:53:13,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-14 23:53:13,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-14 23:53:13,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-14 23:53:13,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-14 23:53:13,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-14 23:53:13,528 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-14 23:53:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-14 23:53:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-14 23:53:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-14 23:53:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-14 23:53:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-14 23:53:13,529 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-14 23:53:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-14 23:53:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-14 23:53:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-14 23:53:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-14 23:53:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-14 23:53:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-14 23:53:13,530 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-14 23:53:13,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-14 23:53:13,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-14 23:53:13,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-14 23:53:13,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-14 23:53:13,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-14 23:53:13,531 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-14 23:53:13,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-14 23:53:13,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-14 23:53:13,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-14 23:53:13,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-14 23:53:13,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-14 23:53:13,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-14 23:53:13,532 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-14 23:53:13,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-14 23:53:13,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-14 23:53:13,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-14 23:53:13,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-14 23:53:13,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-14 23:53:13,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-14 23:53:13,533 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-14 23:53:13,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-14 23:53:13,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-14 23:53:13,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-14 23:53:13,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-14 23:53:13,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-14 23:53:13,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-14 23:53:13,534 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-14 23:53:13,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-14 23:53:13,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-14 23:53:13,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-14 23:53:13,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-14 23:53:13,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-14 23:53:13,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-14 23:53:13,535 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-14 23:53:13,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-14 23:53:13,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-14 23:53:13,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-14 23:53:13,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-14 23:53:13,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-14 23:53:13,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-14 23:53:13,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-14 23:53:13,536 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-14 23:53:13,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-14 23:53:13,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-14 23:53:13,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-14 23:53:13,537 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-14 23:53:13,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-14 23:53:13,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-14 23:53:13,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-14 23:53:13,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-14 23:53:13,752 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-14 23:53:13,752 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-14 23:53:13,753 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-14 23:53:14,119 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 554 ms to try 5000 different iterations.  Found a solution that moves 4713 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1776468095243629. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03725324100775318, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056090355197020686, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5236666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03725324100775318, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.173, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23808333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-14 23:53:14,135 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1593500481:82 , srv1938783145:82 , srv390841821:82 , srv1031321149:83 , srv1423536980:83 , srv792437889:83 , srv793568030:83 , srv1370852155:84 , srv1393746149:84 , srv1690659430:84 , srv1703867815:84 , srv638407791:84 , srv658801312:84 , srv715767378:84 , srv1180094587:85 , srv1188619468:85 , srv1466755558:85 , srv150026344:85 , srv1526379784:85 , srv1605385265:85 , srv1624585469:85 , srv1678447537:85 , srv18374836:85 , srv202530211:85 , srv2046686948:85 , srv2056214813:85 , srv321385986:85 , srv343503108:85 , srv408801329:85 , srv473917378:85 , srv630692188:85 , srv696092676:85 , srv783173800:85 , srv822540784:85 , srv858118781:85 , srv910172570:85 , srv1062248711:86 , srv1082824981:86 , srv1417116795:86 , srv1458447263:86 , srv1520384978:86 , srv1551447105:86 , srv1644514596:86 , srv1712785215:86 , srv309801858:86 , srv500455395:86 , srv519165520:86 , srv554854346:86 , srv594664804:86 , srv602912654:86 , srv61897985:86 , srv680301615:86 , srv688372241:86 , srv758616378:86 , srv818635884:86 , srv1147607115:87 , srv1430988569:87 , srv1436882930:87 , srv1505827474:87 , srv1584577667:87 , srv1980849281:87 , srv2041105836:87 , srv419468499:87 , srv593784087:87 , srv614049239:87 , srv1027846200:88 , srv1263581844:88 , srv1432405231:88 , srv1660657710:88 , srv1664651676:88 , srv177220054:88 , srv1794856270:88 , srv1901633749:88 , srv1937062381:88 , srv2065558605:88 , srv2127561277:88 , srv339772613:88 , srv36446101:88 , srv404956531:88 , srv737162939:88 , srv740928734:88 , srv1451429348:89 , srv1553395038:89 , srv238557422:89 , srv554698718:89 , srv70315645:89 , srv783802171:89 , srv866816453:89 , srv975276356:89 , srv1207042735:90 , srv1813513855:90 , srv756331166:90 , srv2080711270:91 , srv588453639:91 , srv1010994268:92 , srv965055491:92 , srv1869189048:93 , srv1982631872:94 , srv1033366048:96 , srv848322282:421 }
2023-09-14 23:53:14,141 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-14 23:53:14,142 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv858118781=95, srv1180094587=7, srv758616378=87, srv658801312=78, srv70315645=82, srv61897985=75, srv1451429348=18, srv822540784=93, srv756331166=86, srv1458447263=19, srv1432405231=16, srv2127561277=54, srv1010994268=0, srv1436882930=17, srv737162939=84, srv593784087=71, srv1938783145=45, srv783173800=88, srv1551447105=25, srv630692188=76, srv588453639=70, srv404956531=62, srv1393746149=12, srv2065558605=52, srv1937062381=44, srv1370852155=11, srv1703867815=36, srv910172570=97, srv177220054=38, srv408801329=63, srv1553395038=26, srv554698718=68, srv1644514596=31, srv1033366048=3, srv150026344=21, srv1605385265=29, srv975276356=99, srv1660657710=32, srv2056214813=51, srv390841821=61, srv339772613=58, srv419468499=64, srv1147607115=6, srv1690659430=35, srv473917378=65, srv1466755558=20, srv519165520=67, srv309801858=56, srv1980849281=46, srv715767378=83, srv793568030=91, srv594664804=72, srv36446101=60, srv500455395=66, srv1263581844=10, srv638407791=77, srv1869189048=42, srv1712785215=37, srv2080711270=53, srv554854346=69, srv1031321149=2, srv1982631872=47, srv1027846200=1, srv2041105836=49, srv18374836=41, srv602912654=73, srv783802171=89, srv1188619468=8, srv343503108=59, srv1062248711=4, srv238557422=55, srv202530211=48, srv321385986=57, srv1593500481=28, srv1901633749=43, srv1430988569=15, srv965055491=98, srv848322282=94, srv1207042735=9, srv1520384978=23, srv1526379784=24, srv866816453=96, srv1794856270=39, srv1664651676=33, srv1624585469=30, srv818635884=92, srv680301615=79, srv614049239=74, srv1584577667=27, srv792437889=90, srv1417116795=13, srv696092676=81, srv1082824981=5, srv1678447537=34, srv740928734=85, srv2046686948=50, srv688372241=80, srv1813513855=40, srv1423536980=14, srv1505827474=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-14 23:53:14,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-14 23:53:14,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-14 23:53:14,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-14 23:53:14,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-14 23:53:14,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-14 23:53:14,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-14 23:53:14,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-14 23:53:14,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-14 23:53:14,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-14 23:53:14,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-14 23:53:14,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-14 23:53:14,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-14 23:53:14,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-14 23:53:14,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-14 23:53:14,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-14 23:53:14,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-14 23:53:14,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-14 23:53:14,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-14 23:53:14,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-14 23:53:14,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-14 23:53:14,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-14 23:53:14,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-14 23:53:14,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-14 23:53:14,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-14 23:53:14,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-14 23:53:14,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-14 23:53:14,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-14 23:53:14,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-14 23:53:14,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-14 23:53:14,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-14 23:53:14,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-14 23:53:14,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-14 23:53:14,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-14 23:53:14,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-14 23:53:14,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-14 23:53:14,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-14 23:53:14,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-14 23:53:14,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-14 23:53:14,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-14 23:53:14,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-14 23:53:14,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-14 23:53:14,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-14 23:53:14,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-14 23:53:14,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-14 23:53:14,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-14 23:53:14,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-14 23:53:14,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-14 23:53:14,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-14 23:53:14,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-14 23:53:14,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-14 23:53:14,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-14 23:53:14,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-14 23:53:14,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-14 23:53:14,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-14 23:53:14,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-14 23:53:14,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-14 23:53:14,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-14 23:53:14,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-14 23:53:14,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-14 23:53:14,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-14 23:53:14,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-14 23:53:14,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-14 23:53:14,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-14 23:53:14,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-14 23:53:14,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-14 23:53:14,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-14 23:53:14,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-14 23:53:14,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-14 23:53:14,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-14 23:53:14,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-14 23:53:14,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-14 23:53:14,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-14 23:53:14,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-14 23:53:14,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-14 23:53:14,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-14 23:53:14,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-14 23:53:14,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-14 23:53:14,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-14 23:53:14,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-14 23:53:14,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-14 23:53:14,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-14 23:53:14,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-14 23:53:14,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-14 23:53:14,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-14 23:53:14,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-14 23:53:14,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-14 23:53:14,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-14 23:53:14,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-14 23:53:14,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-14 23:53:14,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-14 23:53:14,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-14 23:53:14,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-14 23:53:14,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-14 23:53:14,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-14 23:53:14,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-14 23:53:14,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-14 23:53:14,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-14 23:53:14,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-14 23:53:14,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-14 23:53:14,172 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-14 23:53:14,173 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-14 23:53:14,305 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03725324100775318, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056090355197020686, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03725324100775318, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.173, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23808333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-14 23:53:14,305 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-14 23:53:14,306 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17761380839301555, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03725324100775318, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.056090355197020686, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03725324100775318, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.173, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23808333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-14 23:53:14,607 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 415 ms to try 5000 different iterations.  Found a solution that moves 2614 regions; Going from a computed imbalance of 0.17761380839301555 to a new imbalance of 0.014410371907793339. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01914430859785061); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03510446171533232, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29044444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01914430859785061); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042916666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-14 23:53:14,618 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1033366048:76 , srv1869189048:80 , srv1982631872:80 , srv1010994268:81 , srv2080711270:81 , srv588453639:81 , srv1553395038:82 , srv1660657710:82 , srv339772613:82 , srv70315645:82 , srv756331166:82 , srv975276356:82 , srv1147607115:83 , srv1432405231:83 , srv1451429348:83 , srv1813513855:83 , srv1980849281:83 , srv2127561277:83 , srv404956531:83 , srv866816453:83 , srv965055491:83 , srv1027846200:84 , srv1207042735:84 , srv1263581844:84 , srv1430988569:84 , srv1436882930:84 , srv1505827474:84 , srv1794856270:84 , srv1901633749:84 , srv1937062381:84 , srv2041105836:84 , srv2065558605:84 , srv36446101:84 , srv419468499:84 , srv500455395:84 , srv554698718:84 , srv554854346:84 , srv783802171:84 , srv1062248711:85 , srv1526379784:85 , srv1584577667:85 , srv238557422:85 , srv614049239:85 , srv818635884:85 , srv1082824981:86 , srv1180094587:86 , srv1520384978:86 , srv1551447105:86 , srv1664651676:86 , srv177220054:86 , srv202530211:86 , srv309801858:86 , srv321385986:86 , srv593784087:86 , srv594664804:86 , srv658801312:86 , srv688372241:86 , srv822540784:86 , srv1188619468:87 , srv1393746149:87 , srv1417116795:87 , srv1466755558:87 , srv1690659430:87 , srv1703867815:87 , srv18374836:87 , srv2046686948:87 , srv2056214813:87 , srv519165520:87 , srv737162939:87 , srv758616378:87 , srv783173800:87 , srv1370852155:88 , srv1423536980:88 , srv1458447263:88 , srv1624585469:88 , srv473917378:88 , srv602912654:88 , srv61897985:88 , srv630692188:88 , srv638407791:88 , srv680301615:88 , srv696092676:88 , srv740928734:88 , srv1031321149:89 , srv150026344:89 , srv1644514596:89 , srv1938783145:89 , srv343503108:89 , srv408801329:89 , srv715767378:89 , srv792437889:89 , srv793568030:89 , srv858118781:89 , srv1593500481:90 , srv1605385265:90 , srv1678447537:90 , srv1712785215:90 , srv390841821:90 , srv910172570:90 , srv848322282:523 }
2023-09-14 23:53:14,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-14 23:53:14,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv858118781=95, srv1180094587=7, srv758616378=87, srv658801312=78, srv70315645=82, srv61897985=75, srv1451429348=18, srv822540784=93, srv756331166=86, srv1458447263=19, srv1432405231=16, srv2127561277=54, srv1010994268=0, srv1436882930=17, srv737162939=84, srv593784087=71, srv1938783145=45, srv783173800=88, srv1551447105=25, srv630692188=76, srv588453639=70, srv404956531=62, srv1393746149=12, srv2065558605=52, srv1937062381=44, srv1370852155=11, srv1703867815=36, srv910172570=97, srv177220054=38, srv408801329=63, srv1553395038=26, srv554698718=68, srv1644514596=31, srv1033366048=3, srv150026344=21, srv1605385265=29, srv975276356=99, srv1660657710=32, srv2056214813=51, srv390841821=61, srv339772613=58, srv419468499=64, srv1147607115=6, srv1690659430=35, srv473917378=65, srv1466755558=20, srv519165520=67, srv309801858=56, srv1980849281=46, srv715767378=83, srv793568030=91, srv594664804=72, srv36446101=60, srv500455395=66, srv1263581844=10, srv638407791=77, srv1869189048=42, srv1712785215=37, srv2080711270=53, srv554854346=69, srv1031321149=2, srv1982631872=47, srv1027846200=1, srv2041105836=49, srv18374836=41, srv602912654=73, srv783802171=89, srv1188619468=8, srv343503108=59, srv1062248711=4, srv238557422=55, srv202530211=48, srv321385986=57, srv1593500481=28, srv1901633749=43, srv1430988569=15, srv965055491=98, srv848322282=94, srv1207042735=9, srv1520384978=23, srv1526379784=24, srv866816453=96, srv1794856270=39, srv1664651676=33, srv1624585469=30, srv818635884=92, srv680301615=79, srv614049239=74, srv1584577667=27, srv792437889=90, srv1417116795=13, srv696092676=81, srv1082824981=5, srv1678447537=34, srv740928734=85, srv2046686948=50, srv688372241=80, srv1813513855=40, srv1423536980=14, srv1505827474=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-14 23:53:14,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-14 23:53:14,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-14 23:53:14,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-14 23:53:14,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-14 23:53:14,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-14 23:53:14,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-14 23:53:14,648 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-14 23:53:14,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-14 23:53:14,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-14 23:53:14,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-14 23:53:14,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-14 23:53:14,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-14 23:53:14,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-14 23:53:14,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-14 23:53:14,649 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-14 23:53:14,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-14 23:53:14,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-14 23:53:14,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-14 23:53:14,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-14 23:53:14,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-14 23:53:14,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-14 23:53:14,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-14 23:53:14,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-14 23:53:14,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-14 23:53:14,650 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-14 23:53:14,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-14 23:53:14,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-14 23:53:14,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-14 23:53:14,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-14 23:53:14,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-14 23:53:14,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-14 23:53:14,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-14 23:53:14,651 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-14 23:53:14,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-14 23:53:14,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-14 23:53:14,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-14 23:53:14,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-14 23:53:14,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-14 23:53:14,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-14 23:53:14,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-14 23:53:14,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-14 23:53:14,652 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-14 23:53:14,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-14 23:53:14,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-14 23:53:14,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-14 23:53:14,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-14 23:53:14,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-14 23:53:14,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-14 23:53:14,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-14 23:53:14,653 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-14 23:53:14,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-14 23:53:14,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-14 23:53:14,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-14 23:53:14,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-14 23:53:14,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-14 23:53:14,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-14 23:53:14,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-14 23:53:14,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-14 23:53:14,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-14 23:53:14,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-14 23:53:14,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-14 23:53:14,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-14 23:53:14,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-14 23:53:14,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-14 23:53:14,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-14 23:53:14,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-14 23:53:14,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-14 23:53:14,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-14 23:53:14,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-14 23:53:14,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-14 23:53:14,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-14 23:53:14,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-14 23:53:14,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-14 23:53:14,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-14 23:53:14,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-14 23:53:14,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-14 23:53:14,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-14 23:53:14,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-14 23:53:14,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-14 23:53:14,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-14 23:53:14,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-14 23:53:14,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-14 23:53:14,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-14 23:53:14,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-14 23:53:14,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-14 23:53:14,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-14 23:53:14,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-14 23:53:14,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-14 23:53:14,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-14 23:53:14,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-14 23:53:14,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-14 23:53:14,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-14 23:53:14,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-14 23:53:14,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-14 23:53:14,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-14 23:53:14,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-14 23:53:14,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-14 23:53:14,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-14 23:53:14,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-14 23:53:14,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-14 23:53:14,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-14 23:53:14,787 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01914430859785061); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03510446171533232, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01914430859785061); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042916666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-14 23:53:14,787 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-14 23:53:14,787 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014392068288582242, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.01914430859785061); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03510446171533232, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.01914430859785061); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042916666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-14 23:53:15,145 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 471 ms to try 5000 different iterations.  Found a solution that moves 1004 regions; Going from a computed imbalance of 0.014392068288582242 to a new imbalance of 7.273632658341098E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005102759975884884); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006231594672802901); MoveCostFunction : (multiplier=7.0, imbalance=0.11155555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005102759975884884); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-14 23:53:15,147 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1605385265:82 , srv1712785215:82 , srv740928734:82 , srv910172570:82 , srv965055491:82 , srv1644514596:83 , srv1678447537:83 , srv177220054:83 , srv602912654:83 , srv61897985:83 , srv737162939:83 , srv1010994268:84 , srv1207042735:84 , srv1458447263:84 , srv150026344:84 , srv1593500481:84 , srv1664651676:84 , srv18374836:84 , srv1982631872:84 , srv238557422:84 , srv343503108:84 , srv473917378:84 , srv680301615:84 , srv696092676:84 , srv758616378:84 , srv1031321149:85 , srv1082824981:85 , srv1180094587:85 , srv1263581844:85 , srv1417116795:85 , srv1451429348:85 , srv1466755558:85 , srv1551447105:85 , srv1584577667:85 , srv1624585469:85 , srv1813513855:85 , srv1869189048:85 , srv1937062381:85 , srv2046686948:85 , srv2080711270:85 , srv309801858:85 , srv36446101:85 , srv390841821:85 , srv408801329:85 , srv519165520:85 , srv554698718:85 , srv593784087:85 , srv630692188:85 , srv715767378:85 , srv756331166:85 , srv783802171:85 , srv792437889:85 , srv858118781:85 , srv866816453:85 , srv1027846200:86 , srv1033366048:86 , srv1188619468:86 , srv1370852155:86 , srv1393746149:86 , srv1423536980:86 , srv1432405231:86 , srv1520384978:86 , srv1553395038:86 , srv1703867815:86 , srv1794856270:86 , srv1901633749:86 , srv1938783145:86 , srv202530211:86 , srv2056214813:86 , srv2065558605:86 , srv2127561277:86 , srv321385986:86 , srv404956531:86 , srv588453639:86 , srv594664804:86 , srv614049239:86 , srv688372241:86 , srv70315645:86 , srv783173800:86 , srv818635884:86 , srv822540784:86 , srv1062248711:87 , srv1147607115:87 , srv1430988569:87 , srv1436882930:87 , srv1505827474:87 , srv1660657710:87 , srv339772613:87 , srv419468499:87 , srv500455395:87 , srv554854346:87 , srv638407791:87 , srv658801312:87 , srv793568030:87 , srv975276356:87 , srv1526379784:88 , srv1690659430:88 , srv1980849281:88 , srv2041105836:88 , srv848322282:559 }
2023-09-14 23:53:15,151 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-14 23:53:15,152 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv858118781=95, srv1180094587=7, srv758616378=87, srv658801312=78, srv70315645=82, srv61897985=75, srv1451429348=18, srv822540784=93, srv756331166=86, srv1458447263=19, srv1432405231=16, srv2127561277=54, srv1010994268=0, srv1436882930=17, srv737162939=84, srv593784087=71, srv1938783145=45, srv783173800=88, srv1551447105=25, srv630692188=76, srv588453639=70, srv404956531=62, srv1393746149=12, srv2065558605=52, srv1937062381=44, srv1370852155=11, srv1703867815=36, srv910172570=97, srv177220054=38, srv408801329=63, srv1553395038=26, srv554698718=68, srv1644514596=31, srv1033366048=3, srv150026344=21, srv1605385265=29, srv975276356=99, srv1660657710=32, srv2056214813=51, srv390841821=61, srv339772613=58, srv419468499=64, srv1147607115=6, srv1690659430=35, srv473917378=65, srv1466755558=20, srv519165520=67, srv309801858=56, srv1980849281=46, srv715767378=83, srv793568030=91, srv594664804=72, srv36446101=60, srv500455395=66, srv1263581844=10, srv638407791=77, srv1869189048=42, srv1712785215=37, srv2080711270=53, srv554854346=69, srv1031321149=2, srv1982631872=47, srv1027846200=1, srv2041105836=49, srv18374836=41, srv602912654=73, srv783802171=89, srv1188619468=8, srv343503108=59, srv1062248711=4, srv238557422=55, srv202530211=48, srv321385986=57, srv1593500481=28, srv1901633749=43, srv1430988569=15, srv965055491=98, srv848322282=94, srv1207042735=9, srv1520384978=23, srv1526379784=24, srv866816453=96, srv1794856270=39, srv1664651676=33, srv1624585469=30, srv818635884=92, srv680301615=79, srv614049239=74, srv1584577667=27, srv792437889=90, srv1417116795=13, srv696092676=81, srv1082824981=5, srv1678447537=34, srv740928734=85, srv2046686948=50, srv688372241=80, srv1813513855=40, srv1423536980=14, srv1505827474=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-14 23:53:15,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-14 23:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-14 23:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-14 23:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-14 23:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-14 23:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-14 23:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-14 23:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-14 23:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-14 23:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-14 23:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-14 23:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-14 23:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-14 23:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-14 23:53:15,163 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-14 23:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-14 23:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-14 23:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-14 23:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-14 23:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-14 23:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-14 23:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-14 23:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-14 23:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-14 23:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-14 23:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-14 23:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-14 23:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-14 23:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-14 23:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-14 23:53:15,164 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-14 23:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-14 23:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-14 23:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-14 23:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-14 23:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-14 23:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-14 23:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-14 23:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-14 23:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-14 23:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-14 23:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-14 23:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-14 23:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-14 23:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-14 23:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-14 23:53:15,165 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-14 23:53:15,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-14 23:53:15,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-14 23:53:15,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-14 23:53:15,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-14 23:53:15,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-14 23:53:15,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-14 23:53:15,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-14 23:53:15,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-14 23:53:15,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-14 23:53:15,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-14 23:53:15,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-14 23:53:15,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-14 23:53:15,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-14 23:53:15,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-14 23:53:15,166 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-14 23:53:15,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-14 23:53:15,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-14 23:53:15,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-14 23:53:15,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-14 23:53:15,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-14 23:53:15,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-14 23:53:15,169 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-14 23:53:15,284 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005102759975884884); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006231594672802901); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005102759975884884); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-14 23:53:15,284 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-09-14 23:53:15,285 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.203331075755245E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.005102759975884884); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006231594672802901); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.005102759975884884); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004083333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-09-14 23:53:15,759 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 580 ms to try 5000 different iterations.  Found a solution that moves 231 regions; Going from a computed imbalance of 7.203331075755245E-4 to a new imbalance of 1.8056241996884487E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.025666666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-14 23:53:15,760 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1188619468:83 , srv1678447537:83 , srv1690659430:83 , srv2041105836:83 , srv408801329:83 , srv638407791:83 , srv793568030:83 , srv858118781:83 , srv1027846200:84 , srv1033366048:84 , srv1062248711:84 , srv1207042735:84 , srv1370852155:84 , srv1417116795:84 , srv1430988569:84 , srv1436882930:84 , srv1458447263:84 , srv150026344:84 , srv1505827474:84 , srv1520384978:84 , srv1526379784:84 , srv1624585469:84 , srv1644514596:84 , srv1712785215:84 , srv1794856270:84 , srv1813513855:84 , srv1869189048:84 , srv1980849281:84 , srv2056214813:84 , srv2065558605:84 , srv238557422:84 , srv309801858:84 , srv339772613:84 , srv419468499:84 , srv519165520:84 , srv554698718:84 , srv588453639:84 , srv593784087:84 , srv594664804:84 , srv614049239:84 , srv630692188:84 , srv680301615:84 , srv688372241:84 , srv715767378:84 , srv737162939:84 , srv740928734:84 , srv756331166:84 , srv783173800:84 , srv783802171:84 , srv792437889:84 , srv975276356:84 , srv1010994268:85 , srv1031321149:85 , srv1082824981:85 , srv1147607115:85 , srv1263581844:85 , srv1393746149:85 , srv1423536980:85 , srv1432405231:85 , srv1451429348:85 , srv1466755558:85 , srv1551447105:85 , srv1553395038:85 , srv1584577667:85 , srv1605385265:85 , srv1660657710:85 , srv1664651676:85 , srv1703867815:85 , srv177220054:85 , srv1901633749:85 , srv1937062381:85 , srv1938783145:85 , srv1982631872:85 , srv202530211:85 , srv2046686948:85 , srv2080711270:85 , srv2127561277:85 , srv321385986:85 , srv343503108:85 , srv36446101:85 , srv390841821:85 , srv404956531:85 , srv473917378:85 , srv500455395:85 , srv554854346:85 , srv602912654:85 , srv658801312:85 , srv696092676:85 , srv70315645:85 , srv758616378:85 , srv818635884:85 , srv866816453:85 , srv910172570:85 , srv965055491:85 , srv1180094587:86 , srv1593500481:86 , srv18374836:86 , srv61897985:86 , srv822540784:86 , srv848322282:639 }
2023-09-14 23:53:15,765 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-14 23:53:15,766 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv858118781=95, srv1180094587=7, srv758616378=87, srv658801312=78, srv70315645=82, srv61897985=75, srv1451429348=18, srv822540784=93, srv756331166=86, srv1458447263=19, srv1432405231=16, srv2127561277=54, srv1010994268=0, srv1436882930=17, srv737162939=84, srv593784087=71, srv1938783145=45, srv783173800=88, srv1551447105=25, srv630692188=76, srv588453639=70, srv404956531=62, srv1393746149=12, srv2065558605=52, srv1937062381=44, srv1370852155=11, srv1703867815=36, srv910172570=97, srv177220054=38, srv408801329=63, srv1553395038=26, srv554698718=68, srv1644514596=31, srv1033366048=3, srv150026344=21, srv1605385265=29, srv975276356=99, srv1660657710=32, srv2056214813=51, srv390841821=61, srv339772613=58, srv419468499=64, srv1147607115=6, srv1690659430=35, srv473917378=65, srv1466755558=20, srv519165520=67, srv309801858=56, srv1980849281=46, srv715767378=83, srv793568030=91, srv594664804=72, srv36446101=60, srv500455395=66, srv1263581844=10, srv638407791=77, srv1869189048=42, srv1712785215=37, srv2080711270=53, srv554854346=69, srv1031321149=2, srv1982631872=47, srv1027846200=1, srv2041105836=49, srv18374836=41, srv602912654=73, srv783802171=89, srv1188619468=8, srv343503108=59, srv1062248711=4, srv238557422=55, srv202530211=48, srv321385986=57, srv1593500481=28, srv1901633749=43, srv1430988569=15, srv965055491=98, srv848322282=94, srv1207042735=9, srv1520384978=23, srv1526379784=24, srv866816453=96, srv1794856270=39, srv1664651676=33, srv1624585469=30, srv818635884=92, srv680301615=79, srv614049239=74, srv1584577667=27, srv792437889=90, srv1417116795=13, srv696092676=81, srv1082824981=5, srv1678447537=34, srv740928734=85, srv2046686948=50, srv688372241=80, srv1813513855=40, srv1423536980=14, srv1505827474=22} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-14 23:53:15,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-14 23:53:15,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-14 23:53:15,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-14 23:53:15,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-14 23:53:15,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-14 23:53:15,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-14 23:53:15,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-14 23:53:15,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-14 23:53:15,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-14 23:53:15,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-14 23:53:15,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-14 23:53:15,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-14 23:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-14 23:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-14 23:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-14 23:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-14 23:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-14 23:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-14 23:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-14 23:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-14 23:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-14 23:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-14 23:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-14 23:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-14 23:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-14 23:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-14 23:53:15,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-14 23:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-14 23:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-14 23:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-14 23:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-09-14 23:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-09-14 23:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-09-14 23:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-09-14 23:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-09-14 23:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-09-14 23:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-09-14 23:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-09-14 23:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-09-14 23:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-09-14 23:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-09-14 23:53:15,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-09-14 23:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-09-14 23:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-09-14 23:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-09-14 23:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-09-14 23:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-09-14 23:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-09-14 23:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-09-14 23:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-09-14 23:53:15,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-09-14 23:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-09-14 23:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-09-14 23:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-09-14 23:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-09-14 23:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-09-14 23:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-09-14 23:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-09-14 23:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-09-14 23:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-09-14 23:53:15,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-09-14 23:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-09-14 23:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-09-14 23:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-09-14 23:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-09-14 23:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-09-14 23:53:15,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-09-14 23:53:15,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-09-14 23:53:15,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-09-14 23:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-09-14 23:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-09-14 23:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-09-14 23:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-09-14 23:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-09-14 23:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-09-14 23:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-09-14 23:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-09-14 23:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-09-14 23:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-09-14 23:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-09-14 23:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-09-14 23:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-09-14 23:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-09-14 23:53:15,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-09-14 23:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-09-14 23:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-09-14 23:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-09-14 23:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-09-14 23:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-09-14 23:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-09-14 23:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-09-14 23:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-09-14 23:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-09-14 23:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-09-14 23:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-09-14 23:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-09-14 23:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-09-14 23:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-09-14 23:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-09-14 23:53:15,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-09-14 23:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-09-14 23:53:15,926 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.643874542543705E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.868392354571063E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.868392354571063E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-14 23:53:15,927 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1188619468:83 , srv1678447537:83 , srv1690659430:83 , srv2041105836:83 , srv408801329:83 , srv638407791:83 , srv793568030:83 , srv858118781:83 , srv1027846200:84 , srv1033366048:84 , srv1062248711:84 , srv1207042735:84 , srv1370852155:84 , srv1417116795:84 , srv1430988569:84 , srv1436882930:84 , srv1458447263:84 , srv150026344:84 , srv1505827474:84 , srv1520384978:84 , srv1526379784:84 , srv1624585469:84 , srv1644514596:84 , srv1712785215:84 , srv1794856270:84 , srv1813513855:84 , srv1869189048:84 , srv1980849281:84 , srv2056214813:84 , srv2065558605:84 , srv238557422:84 , srv309801858:84 , srv339772613:84 , srv419468499:84 , srv519165520:84 , srv554698718:84 , srv588453639:84 , srv593784087:84 , srv594664804:84 , srv614049239:84 , srv630692188:84 , srv680301615:84 , srv688372241:84 , srv715767378:84 , srv737162939:84 , srv740928734:84 , srv756331166:84 , srv783173800:84 , srv783802171:84 , srv792437889:84 , srv975276356:84 , srv1010994268:85 , srv1031321149:85 , srv1082824981:85 , srv1147607115:85 , srv1263581844:85 , srv1393746149:85 , srv1423536980:85 , srv1432405231:85 , srv1451429348:85 , srv1466755558:85 , srv1551447105:85 , srv1553395038:85 , srv1584577667:85 , srv1605385265:85 , srv1660657710:85 , srv1664651676:85 , srv1703867815:85 , srv177220054:85 , srv1901633749:85 , srv1937062381:85 , srv1938783145:85 , srv1982631872:85 , srv202530211:85 , srv2046686948:85 , srv2080711270:85 , srv2127561277:85 , srv321385986:85 , srv343503108:85 , srv36446101:85 , srv390841821:85 , srv404956531:85 , srv473917378:85 , srv500455395:85 , srv554854346:85 , srv602912654:85 , srv658801312:85 , srv696092676:85 , srv70315645:85 , srv758616378:85 , srv818635884:85 , srv866816453:85 , srv910172570:85 , srv965055491:85 , srv1180094587:86 , srv1593500481:86 , srv18374836:86 , srv61897985:86 , srv822540784:86 , srv848322282:639 }
2023-09-14 23:53:15,968 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=337 (was 337), ProcessCount=169 (was 169), AvailableMemoryMB=9465 (was 10267)
2023-09-14 23:53:15,978 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=337, ProcessCount=169, AvailableMemoryMB=9465
2023-09-14 23:53:15,979 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-09-14 23:53:15,979 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-09-14 23:53:15,992 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1025555827:84 , srv1029461587:84 , srv1036722340:84 , srv1089808848:84 , srv1159345057:84 , srv1363958712:84 , srv1403533725:84 , srv1423473470:84 , srv1459160583:84 , srv1524777067:84 , srv1589958398:84 , srv1652654962:84 , srv1727370783:84 , srv1731192889:84 , srv1866756453:84 , srv1984237309:84 , srv2002343821:84 , srv207719368:84 , srv2132695362:84 , srv274869536:84 , srv30081415:84 , srv352658565:84 , srv354789764:84 , srv412948589:84 , srv522168253:84 , srv591505570:84 , srv851498469:84 , srv88601224:84 , srv958854235:84 , srv854181700:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-09-14 23:53:15,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-14 23:53:15,994 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1731192889=13, srv1524777067=9, srv1459160583=8, srv2002343821=16, srv591505570=25, srv1089808848=3, srv1029461587=1, srv1866756453=14, srv274869536=19, srv1423473470=7, srv352658565=21, srv522168253=24, srv851498469=26, srv412948589=23, srv1403533725=6, srv1652654962=11, srv1363958712=5, srv207719368=17, srv958854235=29, srv1984237309=15, srv854181700=27, srv88601224=28, srv1159345057=4, srv1727370783=12, srv30081415=20, srv1036722340=2, srv354789764=22, srv2132695362=18, srv1589958398=10, srv1025555827=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-14 23:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-14 23:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-14 23:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-14 23:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-14 23:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-14 23:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-14 23:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-14 23:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-14 23:53:16,007 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-14 23:53:16,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-14 23:53:16,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-14 23:53:16,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-14 23:53:16,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-14 23:53:16,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-14 23:53:16,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-14 23:53:16,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-14 23:53:16,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-14 23:53:16,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-14 23:53:16,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-14 23:53:16,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-14 23:53:16,008 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-14 23:53:16,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-14 23:53:16,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-14 23:53:16,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-14 23:53:16,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-14 23:53:16,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-14 23:53:16,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-14 23:53:16,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-14 23:53:16,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-14 23:53:16,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-14 23:53:16,009 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-14 23:53:16,039 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0950984194853828, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-14 23:53:16,039 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912492695278831, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0950984194853828, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-09-14 23:55:16,013 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5444740 different iterations.  Found a solution that moves 1867 regions; Going from a computed imbalance of 0.9912492695278831 to a new imbalance of 4.3576711383728133E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6914814814814815, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-14 23:55:16,021 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1025555827:90 , srv1029461587:90 , srv1036722340:90 , srv1089808848:90 , srv1159345057:90 , srv1363958712:90 , srv1403533725:90 , srv1423473470:90 , srv1459160583:90 , srv1524777067:90 , srv1589958398:90 , srv1652654962:90 , srv1727370783:90 , srv1731192889:90 , srv1866756453:90 , srv1984237309:90 , srv2002343821:90 , srv207719368:90 , srv2132695362:90 , srv274869536:90 , srv30081415:90 , srv352658565:90 , srv354789764:90 , srv412948589:90 , srv522168253:90 , srv591505570:90 , srv851498469:90 , srv854181700:90 , srv88601224:90 , srv958854235:90 }
2023-09-14 23:55:16,022 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-09-14 23:55:16,023 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1731192889=13, srv1524777067=9, srv1459160583=8, srv2002343821=16, srv591505570=25, srv1089808848=3, srv1029461587=1, srv1866756453=14, srv274869536=19, srv1423473470=7, srv352658565=21, srv522168253=24, srv851498469=26, srv412948589=23, srv1403533725=6, srv1652654962=11, srv1363958712=5, srv207719368=17, srv958854235=29, srv1984237309=15, srv854181700=27, srv88601224=28, srv1159345057=4, srv1727370783=12, srv30081415=20, srv1036722340=2, srv354789764=22, srv2132695362=18, srv1589958398=10, srv1025555827=0} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-09-14 23:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-09-14 23:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-09-14 23:55:16,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-09-14 23:55:16,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-09-14 23:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-09-14 23:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-09-14 23:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-09-14 23:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-09-14 23:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-09-14 23:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-09-14 23:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-09-14 23:55:16,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-09-14 23:55:16,039 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-09-14 23:55:16,039 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1025555827:90 , srv1029461587:90 , srv1036722340:90 , srv1089808848:90 , srv1159345057:90 , srv1363958712:90 , srv1403533725:90 , srv1423473470:90 , srv1459160583:90 , srv1524777067:90 , srv1589958398:90 , srv1652654962:90 , srv1727370783:90 , srv1731192889:90 , srv1866756453:90 , srv1984237309:90 , srv2002343821:90 , srv207719368:90 , srv2132695362:90 , srv274869536:90 , srv30081415:90 , srv352658565:90 , srv354789764:90 , srv412948589:90 , srv522168253:90 , srv591505570:90 , srv851498469:90 , srv854181700:90 , srv88601224:90 , srv958854235:90 }
2023-09-14 23:55:16,054 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=146 (was 337), ProcessCount=169 (was 169), AvailableMemoryMB=8708 (was 9465)