2024-03-02 01:53:17,841 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-03-02 01:53:18,170 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-03-02 01:53:18,287 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-03-02 01:53:18,359 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-02 01:53:18,368 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-02 01:53:18,386 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=277, ProcessCount=170, AvailableMemoryMB=10269
2024-03-02 01:53:18,388 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-02 01:53:18,389 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-02 01:53:18,526 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000807179:84 , srv1027584769:84 , srv1057188421:84 , srv1066561784:84 , srv1100047204:84 , srv1109782307:84 , srv1111138267:84 , srv1119597093:84 , srv1147696000:84 , srv1150300292:84 , srv115380471:84 , srv1157743116:84 , srv1161077576:84 , srv1171518357:84 , srv1191045713:84 , srv1204549625:84 , srv1210661196:84 , srv1213843800:84 , srv1215252073:84 , srv1221134651:84 , srv122150217:84 , srv1235851470:84 , srv1260666902:84 , srv1295056832:84 , srv1347443490:84 , srv1379759343:84 , srv1416481897:84 , srv1421846551:84 , srv1449537124:84 , srv1460341015:84 , srv1507845262:84 , srv1590999645:84 , srv160889163:84 , srv1625429285:84 , srv1634521368:84 , srv1658567210:84 , srv1686061562:84 , srv1721531087:84 , srv1772938074:84 , srv179136220:84 , srv1833407564:84 , srv1836254134:84 , srv1859361440:84 , srv1863127024:84 , srv190427988:84 , srv1912785164:84 , srv1920506643:84 , srv1971933930:84 , srv1978183626:84 , srv1980909166:84 , srv1983221321:84 , srv1990819571:84 , srv1994419406:84 , srv2048519113:84 , srv2052182277:84 , srv2059159000:84 , srv2087147108:84 , srv2110121532:84 , srv2119968664:84 , srv2144746849:84 , srv216882382:84 , srv219880756:84 , srv224053702:84 , srv226301555:84 , srv231382324:84 , srv235406041:84 , srv261107455:84 , srv268620265:84 , srv274744506:84 , srv286997461:84 , srv308849759:84 , srv33396993:84 , srv364705010:84 , srv486683234:84 , srv498004756:84 , srv530320249:84 , srv535737880:84 , srv566809373:84 , srv586302199:84 , srv594853229:84 , srv60285109:84 , srv606687536:84 , srv65282260:84 , srv656809320:84 , srv672451572:84 , srv722887952:84 , srv726426302:84 , srv744325085:84 , srv74859161:84 , srv792899054:84 , srv87435059:84 , srv879437859:84 , srv916528039:84 , srv944414039:84 , srv955354334:84 , srv972558944:84 , srv980055994:84 , srv985744895:84 , srv994836731:84 , srv111809942:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-03-02 01:53:18,538 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-02 01:53:18,546 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1460341015=30, srv1109782307=5, srv1147696000=9, srv274744506=69, srv1066561784=3, srv122150217=21, srv1295056832=24, srv65282260=83, srv1171518357=14, srv1235851470=22, srv972558944=96, srv1421846551=28, srv1416481897=27, srv286997461=70, srv33396993=72, srv606687536=82, srv1379759343=26, srv586302199=79, srv2048519113=54, srv226301555=64, srv1191045713=15, srv160889163=33, srv985744895=98, srv530320249=76, srv726426302=87, srv216882382=61, srv60285109=81, srv1978183626=49, srv1150300292=10, srv1215252073=19, srv2144746849=60, srv1119597093=8, srv944414039=94, srv1658567210=36, srv566809373=78, srv1221134651=20, srv955354334=95, srv980055994=97, srv994836731=99, srv1772938074=39, srv224053702=63, srv364705010=73, srv1213843800=18, srv179136220=40, srv1971933930=48, srv308849759=71, srv87435059=91, srv1686061562=37, srv1920506643=47, srv879437859=92, srv1000807179=0, srv1210661196=17, srv1057188421=2, srv594853229=80, srv1994419406=53, srv535737880=77, srv2059159000=56, srv115380471=11, srv1347443490=25, srv1859361440=43, srv1204549625=16, srv190427988=45, srv1625429285=34, srv1161077576=13, srv916528039=93, srv1100047204=4, srv1634521368=35, srv1157743116=12, srv261107455=67, srv656809320=84, srv1912785164=46, srv722887952=86, srv672451572=85, srv111809942=7, srv744325085=88, srv1836254134=42, srv1833407564=41, srv1983221321=51, srv1027584769=1, srv74859161=89, srv486683234=74, srv1590999645=32, srv1980909166=50, srv231382324=65, srv1721531087=38, srv219880756=62, srv2087147108=57, srv1260666902=23, srv1449537124=29, srv1863127024=44, srv792899054=90, srv498004756=75, srv235406041=66, srv268620265=68, srv1111138267=6, srv2119968664=59, srv2110121532=58, srv2052182277=55, srv1507845262=31, srv1990819571=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-02 01:53:18,630 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-02 01:53:18,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-02 01:53:18,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-02 01:53:18,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-02 01:53:18,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-02 01:53:18,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-02 01:53:18,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-02 01:53:18,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-02 01:53:18,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-02 01:53:18,631 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-02 01:53:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-02 01:53:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-02 01:53:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-02 01:53:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-02 01:53:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-02 01:53:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-02 01:53:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-02 01:53:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-02 01:53:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-02 01:53:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-02 01:53:18,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-02 01:53:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-02 01:53:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-02 01:53:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-02 01:53:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-02 01:53:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-02 01:53:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-02 01:53:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-02 01:53:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-02 01:53:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-02 01:53:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-02 01:53:18,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-02 01:53:18,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-02 01:53:18,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-02 01:53:18,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-02 01:53:18,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-02 01:53:18,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-02 01:53:18,634 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-02 01:53:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-02 01:53:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-02 01:53:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-02 01:53:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-02 01:53:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-02 01:53:18,635 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-02 01:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-02 01:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-02 01:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-02 01:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-02 01:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-02 01:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-02 01:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-02 01:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-02 01:53:18,636 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-02 01:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-02 01:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-02 01:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-02 01:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-02 01:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-02 01:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-02 01:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-02 01:53:18,637 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-02 01:53:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-02 01:53:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-02 01:53:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-02 01:53:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-02 01:53:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-02 01:53:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-02 01:53:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-02 01:53:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-02 01:53:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-02 01:53:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-02 01:53:18,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-02 01:53:18,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-02 01:53:18,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-02 01:53:18,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-02 01:53:18,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-02 01:53:18,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-02 01:53:18,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-02 01:53:18,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-02 01:53:18,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-02 01:53:18,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-02 01:53:18,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-02 01:53:18,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-02 01:53:18,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-02 01:53:18,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-02 01:53:18,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-02 01:53:18,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-02 01:53:18,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-02 01:53:18,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-02 01:53:18,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-02 01:53:18,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-02 01:53:18,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-02 01:53:18,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-02 01:53:18,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-02 01:53:18,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-02 01:53:18,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-02 01:53:18,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-02 01:53:18,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-02 01:53:18,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-02 01:53:18,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-02 01:53:18,641 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-02 01:53:18,876 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-02 01:53:18,876 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-02 01:53:18,876 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-02 01:53:19,895 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1219 ms to try 5000 different iterations.  Found a solution that moves 4682 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18300221156640806. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040718696209176244, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05882119226888095, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5202222222222223, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040718696209176244, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17816666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24558333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-02 01:53:19,917 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1863127024:81 , srv1421846551:82 , srv594853229:82 , srv879437859:82 , srv1057188421:83 , srv1119597093:83 , srv1161077576:83 , srv1221134651:83 , srv1833407564:83 , srv1066561784:84 , srv1109782307:84 , srv1157743116:84 , srv1686061562:84 , srv179136220:84 , srv1836254134:84 , srv1920506643:84 , srv1990819571:84 , srv2059159000:84 , srv2110121532:84 , srv224053702:84 , srv566809373:84 , srv586302199:84 , srv60285109:84 , srv985744895:84 , srv1000807179:85 , srv1027584769:85 , srv1215252073:85 , srv122150217:85 , srv1379759343:85 , srv1416481897:85 , srv160889163:85 , srv1658567210:85 , srv1971933930:85 , srv1980909166:85 , srv2052182277:85 , srv2087147108:85 , srv268620265:85 , srv498004756:85 , srv656809320:85 , srv722887952:85 , srv1147696000:86 , srv1210661196:86 , srv1449537124:86 , srv1460341015:86 , srv1507845262:86 , srv1590999645:86 , srv1721531087:86 , srv1859361440:86 , srv308849759:86 , srv33396993:86 , srv364705010:86 , srv486683234:86 , srv530320249:86 , srv726426302:86 , srv74859161:86 , srv944414039:86 , srv972558944:86 , srv1111138267:87 , srv115380471:87 , srv1204549625:87 , srv1213843800:87 , srv1295056832:87 , srv2144746849:87 , srv216882382:87 , srv226301555:87 , srv231382324:87 , srv65282260:87 , srv672451572:87 , srv792899054:87 , srv916528039:87 , srv1100047204:88 , srv1235851470:88 , srv1634521368:88 , srv190427988:88 , srv1912785164:88 , srv1983221321:88 , srv2048519113:88 , srv274744506:88 , srv535737880:88 , srv87435059:88 , srv980055994:88 , srv1150300292:89 , srv1625429285:89 , srv1772938074:89 , srv286997461:89 , srv744325085:89 , srv2119968664:90 , srv219880756:90 , srv235406041:90 , srv261107455:90 , srv994836731:90 , srv1171518357:91 , srv1191045713:91 , srv1347443490:91 , srv1978183626:91 , srv1994419406:91 , srv606687536:91 , srv955354334:91 , srv1260666902:93 , srv111809942:452 }
2024-03-02 01:53:19,924 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-02 01:53:19,925 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1460341015=30, srv1109782307=5, srv1147696000=9, srv274744506=69, srv1066561784=3, srv122150217=21, srv1295056832=24, srv65282260=83, srv1171518357=14, srv1235851470=22, srv972558944=96, srv1421846551=28, srv1416481897=27, srv286997461=70, srv33396993=72, srv606687536=82, srv1379759343=26, srv586302199=79, srv2048519113=54, srv226301555=64, srv1191045713=15, srv160889163=33, srv985744895=98, srv530320249=76, srv726426302=87, srv216882382=61, srv60285109=81, srv1978183626=49, srv1150300292=10, srv1215252073=19, srv2144746849=60, srv1119597093=8, srv944414039=94, srv1658567210=36, srv566809373=78, srv1221134651=20, srv955354334=95, srv980055994=97, srv994836731=99, srv1772938074=39, srv224053702=63, srv364705010=73, srv1213843800=18, srv179136220=40, srv1971933930=48, srv308849759=71, srv87435059=91, srv1686061562=37, srv1920506643=47, srv879437859=92, srv1000807179=0, srv1210661196=17, srv1057188421=2, srv594853229=80, srv1994419406=53, srv535737880=77, srv2059159000=56, srv115380471=11, srv1347443490=25, srv1859361440=43, srv1204549625=16, srv190427988=45, srv1625429285=34, srv1161077576=13, srv916528039=93, srv1100047204=4, srv1634521368=35, srv1157743116=12, srv261107455=67, srv656809320=84, srv1912785164=46, srv722887952=86, srv672451572=85, srv111809942=7, srv744325085=88, srv1836254134=42, srv1833407564=41, srv1983221321=51, srv1027584769=1, srv74859161=89, srv486683234=74, srv1590999645=32, srv1980909166=50, srv231382324=65, srv1721531087=38, srv219880756=62, srv2087147108=57, srv1260666902=23, srv1449537124=29, srv1863127024=44, srv792899054=90, srv498004756=75, srv235406041=66, srv268620265=68, srv1111138267=6, srv2119968664=59, srv2110121532=58, srv2052182277=55, srv1507845262=31, srv1990819571=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-02 01:53:19,952 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-02 01:53:19,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-02 01:53:19,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-02 01:53:19,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-02 01:53:19,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-02 01:53:19,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-02 01:53:19,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-02 01:53:19,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-02 01:53:19,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-02 01:53:19,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-02 01:53:19,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-02 01:53:19,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-02 01:53:19,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-02 01:53:19,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-02 01:53:19,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-02 01:53:19,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-02 01:53:19,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-02 01:53:19,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-02 01:53:19,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-02 01:53:19,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-02 01:53:19,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-02 01:53:19,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-02 01:53:19,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-02 01:53:19,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-02 01:53:19,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-02 01:53:19,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-02 01:53:19,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-02 01:53:19,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-02 01:53:19,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-02 01:53:19,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-02 01:53:19,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-02 01:53:19,956 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-02 01:53:19,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-02 01:53:19,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-02 01:53:19,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-02 01:53:19,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-02 01:53:19,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-02 01:53:19,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-02 01:53:19,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-02 01:53:19,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-02 01:53:19,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-02 01:53:19,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-02 01:53:19,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-02 01:53:19,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-02 01:53:19,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-02 01:53:19,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-02 01:53:19,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-02 01:53:19,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-02 01:53:19,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-02 01:53:19,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-02 01:53:19,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-02 01:53:19,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-02 01:53:19,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-02 01:53:19,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-02 01:53:19,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-02 01:53:19,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-02 01:53:19,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-02 01:53:19,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-02 01:53:19,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-02 01:53:19,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-02 01:53:19,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-02 01:53:19,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-02 01:53:19,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-02 01:53:19,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-02 01:53:19,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-02 01:53:19,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-02 01:53:19,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-02 01:53:19,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-02 01:53:19,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-02 01:53:19,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-02 01:53:19,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-02 01:53:19,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-02 01:53:19,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-02 01:53:19,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-02 01:53:19,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-02 01:53:19,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-02 01:53:19,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-02 01:53:19,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-02 01:53:19,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-02 01:53:19,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-02 01:53:19,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-02 01:53:19,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-02 01:53:19,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-02 01:53:19,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-02 01:53:19,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-02 01:53:19,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-02 01:53:19,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-02 01:53:19,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-02 01:53:19,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-02 01:53:19,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-02 01:53:19,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-02 01:53:19,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-02 01:53:19,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-02 01:53:19,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-02 01:53:19,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-02 01:53:19,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-02 01:53:19,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-02 01:53:19,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-02 01:53:19,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-02 01:53:19,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-02 01:53:19,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-02 01:53:20,143 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040718696209176244, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05882119226888095, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040718696209176244, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17816666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24558333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-02 01:53:20,143 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-02 01:53:20,143 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18296942750170017, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.040718696209176244, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05882119226888095, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.040718696209176244, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17816666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24558333333333332, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-02 01:53:20,461 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 467 ms to try 5000 different iterations.  Found a solution that moves 2628 regions; Going from a computed imbalance of 0.18296942750170017 to a new imbalance of 0.014954641922515798. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02418699236832506); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04059877315380004, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.292, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02418699236832506); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04841666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-02 01:53:20,476 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1260666902:76 , srv219880756:78 , srv1191045713:79 , srv994836731:80 , srv1150300292:81 , srv1625429285:81 , srv1772938074:81 , srv1978183626:81 , srv1994419406:81 , srv261107455:81 , srv286997461:81 , srv1347443490:82 , srv2119968664:82 , srv235406041:82 , srv955354334:82 , srv1634521368:83 , srv1912785164:83 , srv1983221321:83 , srv535737880:83 , srv744325085:83 , srv87435059:83 , srv916528039:83 , srv980055994:83 , srv1100047204:84 , srv1111138267:84 , srv1171518357:84 , srv1213843800:84 , srv190427988:84 , srv2144746849:84 , srv606687536:84 , srv792899054:84 , srv1000807179:85 , srv1027584769:85 , srv1147696000:85 , srv115380471:85 , srv1204549625:85 , srv1235851470:85 , srv1507845262:85 , srv1590999645:85 , srv160889163:85 , srv1721531087:85 , srv1971933930:85 , srv1980909166:85 , srv216882382:85 , srv268620265:85 , srv274744506:85 , srv364705010:85 , srv498004756:85 , srv944414039:85 , srv1215252073:86 , srv1460341015:86 , srv2052182277:86 , srv2110121532:86 , srv231382324:86 , srv308849759:86 , srv33396993:86 , srv486683234:86 , srv65282260:86 , srv656809320:86 , srv672451572:86 , srv726426302:86 , srv972558944:86 , srv1066561784:87 , srv1157743116:87 , srv1210661196:87 , srv122150217:87 , srv1379759343:87 , srv1416481897:87 , srv1449537124:87 , srv1686061562:87 , srv1859361440:87 , srv1990819571:87 , srv2048519113:87 , srv60285109:87 , srv722887952:87 , srv74859161:87 , srv1221134651:88 , srv1295056832:88 , srv224053702:88 , srv530320249:88 , srv566809373:88 , srv985744895:88 , srv1421846551:89 , srv179136220:89 , srv1833407564:89 , srv1920506643:89 , srv2059159000:89 , srv226301555:89 , srv586302199:89 , srv594853229:89 , srv1057188421:90 , srv1109782307:90 , srv1119597093:90 , srv1658567210:90 , srv1863127024:90 , srv2087147108:90 , srv1161077576:91 , srv1836254134:91 , srv879437859:91 , srv111809942:537 }
2024-03-02 01:53:20,484 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-02 01:53:20,485 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1460341015=30, srv1109782307=5, srv1147696000=9, srv274744506=69, srv1066561784=3, srv122150217=21, srv1295056832=24, srv65282260=83, srv1171518357=14, srv1235851470=22, srv972558944=96, srv1421846551=28, srv1416481897=27, srv286997461=70, srv33396993=72, srv606687536=82, srv1379759343=26, srv586302199=79, srv2048519113=54, srv226301555=64, srv1191045713=15, srv160889163=33, srv985744895=98, srv530320249=76, srv726426302=87, srv216882382=61, srv60285109=81, srv1978183626=49, srv1150300292=10, srv1215252073=19, srv2144746849=60, srv1119597093=8, srv944414039=94, srv1658567210=36, srv566809373=78, srv1221134651=20, srv955354334=95, srv980055994=97, srv994836731=99, srv1772938074=39, srv224053702=63, srv364705010=73, srv1213843800=18, srv179136220=40, srv1971933930=48, srv308849759=71, srv87435059=91, srv1686061562=37, srv1920506643=47, srv879437859=92, srv1000807179=0, srv1210661196=17, srv1057188421=2, srv594853229=80, srv1994419406=53, srv535737880=77, srv2059159000=56, srv115380471=11, srv1347443490=25, srv1859361440=43, srv1204549625=16, srv190427988=45, srv1625429285=34, srv1161077576=13, srv916528039=93, srv1100047204=4, srv1634521368=35, srv1157743116=12, srv261107455=67, srv656809320=84, srv1912785164=46, srv722887952=86, srv672451572=85, srv111809942=7, srv744325085=88, srv1836254134=42, srv1833407564=41, srv1983221321=51, srv1027584769=1, srv74859161=89, srv486683234=74, srv1590999645=32, srv1980909166=50, srv231382324=65, srv1721531087=38, srv219880756=62, srv2087147108=57, srv1260666902=23, srv1449537124=29, srv1863127024=44, srv792899054=90, srv498004756=75, srv235406041=66, srv268620265=68, srv1111138267=6, srv2119968664=59, srv2110121532=58, srv2052182277=55, srv1507845262=31, srv1990819571=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-02 01:53:20,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-02 01:53:20,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-02 01:53:20,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-02 01:53:20,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-02 01:53:20,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-02 01:53:20,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-02 01:53:20,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-02 01:53:20,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-02 01:53:20,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-02 01:53:20,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-02 01:53:20,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-02 01:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-02 01:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-02 01:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-02 01:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-02 01:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-02 01:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-02 01:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-02 01:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-02 01:53:20,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-02 01:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-02 01:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-02 01:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-02 01:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-02 01:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-02 01:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-02 01:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-02 01:53:20,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-02 01:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-02 01:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-02 01:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-02 01:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-02 01:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-02 01:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-02 01:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-02 01:53:20,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-02 01:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-02 01:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-02 01:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-02 01:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-02 01:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-02 01:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-02 01:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-02 01:53:20,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-02 01:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-02 01:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-02 01:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-02 01:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-02 01:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-02 01:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-02 01:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-02 01:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-02 01:53:20,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-02 01:53:20,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-02 01:53:20,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-02 01:53:20,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-02 01:53:20,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-02 01:53:20,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-02 01:53:20,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-02 01:53:20,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-02 01:53:20,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-02 01:53:20,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-02 01:53:20,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-02 01:53:20,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-02 01:53:20,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-02 01:53:20,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-02 01:53:20,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-02 01:53:20,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-02 01:53:20,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-02 01:53:20,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-02 01:53:20,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-02 01:53:20,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-02 01:53:20,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-02 01:53:20,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-02 01:53:20,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-02 01:53:20,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-02 01:53:20,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-02 01:53:20,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-02 01:53:20,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-02 01:53:20,513 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-02 01:53:20,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-02 01:53:20,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-02 01:53:20,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-02 01:53:20,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-02 01:53:20,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-02 01:53:20,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-02 01:53:20,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-02 01:53:20,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-02 01:53:20,514 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-02 01:53:20,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-02 01:53:20,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-02 01:53:20,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-02 01:53:20,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-02 01:53:20,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-02 01:53:20,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-02 01:53:20,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-02 01:53:20,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-02 01:53:20,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-02 01:53:20,515 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-02 01:53:20,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-02 01:53:20,516 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-02 01:53:20,649 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02418699236832506); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04059877315380004, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02418699236832506); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04841666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-02 01:53:20,650 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-02 01:53:20,650 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014936240273209462, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02418699236832506); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04059877315380004, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02418699236832506); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011416666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04841666666666666, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-02 01:53:21,021 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 479 ms to try 5000 different iterations.  Found a solution that moves 1018 regions; Going from a computed imbalance of 0.014936240273209462 to a new imbalance of 7.02332201461632E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0065610792988867375); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00908866414480969); MoveCostFunction : (multiplier=7.0, imbalance=0.1131111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0065610792988867375); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-02 01:53:21,023 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1161077576:82 , srv1171518357:82 , srv1836254134:82 , srv2048519113:82 , srv2087147108:82 , srv226301555:82 , srv1057188421:83 , srv1109782307:83 , srv1210661196:83 , srv1295056832:83 , srv1658567210:83 , srv586302199:83 , srv606687536:83 , srv1235851470:84 , srv1449537124:84 , srv190427988:84 , srv1920506643:84 , srv2059159000:84 , srv274744506:84 , srv672451572:84 , srv74859161:84 , srv955354334:84 , srv1100047204:85 , srv1204549625:85 , srv122150217:85 , srv1347443490:85 , srv1379759343:85 , srv1416481897:85 , srv179136220:85 , srv1833407564:85 , srv1859361440:85 , srv1994419406:85 , srv224053702:85 , srv231382324:85 , srv235406041:85 , srv530320249:85 , srv65282260:85 , srv722887952:85 , srv726426302:85 , srv972558944:85 , srv985744895:85 , srv1111138267:86 , srv1119597093:86 , srv115380471:86 , srv1213843800:86 , srv1215252073:86 , srv1421846551:86 , srv1460341015:86 , srv1507845262:86 , srv1590999645:86 , srv1634521368:86 , srv1721531087:86 , srv1863127024:86 , srv1912785164:86 , srv1978183626:86 , srv1983221321:86 , srv216882382:86 , srv261107455:86 , srv33396993:86 , srv364705010:86 , srv486683234:86 , srv535737880:86 , srv594853229:86 , srv744325085:86 , srv792899054:86 , srv879437859:86 , srv1000807179:87 , srv1027584769:87 , srv1066561784:87 , srv1147696000:87 , srv1150300292:87 , srv1157743116:87 , srv1260666902:87 , srv160889163:87 , srv1625429285:87 , srv1686061562:87 , srv1772938074:87 , srv2110121532:87 , srv2119968664:87 , srv2144746849:87 , srv268620265:87 , srv286997461:87 , srv308849759:87 , srv498004756:87 , srv566809373:87 , srv60285109:87 , srv656809320:87 , srv87435059:87 , srv916528039:87 , srv944414039:87 , srv980055994:87 , srv994836731:87 , srv1191045713:88 , srv1221134651:88 , srv1971933930:88 , srv1980909166:88 , srv1990819571:88 , srv2052182277:88 , srv219880756:89 , srv111809942:527 }
2024-03-02 01:53:21,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-02 01:53:21,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1460341015=30, srv1109782307=5, srv1147696000=9, srv274744506=69, srv1066561784=3, srv122150217=21, srv1295056832=24, srv65282260=83, srv1171518357=14, srv1235851470=22, srv972558944=96, srv1421846551=28, srv1416481897=27, srv286997461=70, srv33396993=72, srv606687536=82, srv1379759343=26, srv586302199=79, srv2048519113=54, srv226301555=64, srv1191045713=15, srv160889163=33, srv985744895=98, srv530320249=76, srv726426302=87, srv216882382=61, srv60285109=81, srv1978183626=49, srv1150300292=10, srv1215252073=19, srv2144746849=60, srv1119597093=8, srv944414039=94, srv1658567210=36, srv566809373=78, srv1221134651=20, srv955354334=95, srv980055994=97, srv994836731=99, srv1772938074=39, srv224053702=63, srv364705010=73, srv1213843800=18, srv179136220=40, srv1971933930=48, srv308849759=71, srv87435059=91, srv1686061562=37, srv1920506643=47, srv879437859=92, srv1000807179=0, srv1210661196=17, srv1057188421=2, srv594853229=80, srv1994419406=53, srv535737880=77, srv2059159000=56, srv115380471=11, srv1347443490=25, srv1859361440=43, srv1204549625=16, srv190427988=45, srv1625429285=34, srv1161077576=13, srv916528039=93, srv1100047204=4, srv1634521368=35, srv1157743116=12, srv261107455=67, srv656809320=84, srv1912785164=46, srv722887952=86, srv672451572=85, srv111809942=7, srv744325085=88, srv1836254134=42, srv1833407564=41, srv1983221321=51, srv1027584769=1, srv74859161=89, srv486683234=74, srv1590999645=32, srv1980909166=50, srv231382324=65, srv1721531087=38, srv219880756=62, srv2087147108=57, srv1260666902=23, srv1449537124=29, srv1863127024=44, srv792899054=90, srv498004756=75, srv235406041=66, srv268620265=68, srv1111138267=6, srv2119968664=59, srv2110121532=58, srv2052182277=55, srv1507845262=31, srv1990819571=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-02 01:53:21,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-02 01:53:21,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-02 01:53:21,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-02 01:53:21,039 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-02 01:53:21,040 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-02 01:53:21,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-02 01:53:21,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-02 01:53:21,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-02 01:53:21,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-02 01:53:21,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-02 01:53:21,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-02 01:53:21,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-02 01:53:21,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-02 01:53:21,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-02 01:53:21,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-02 01:53:21,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-02 01:53:21,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-02 01:53:21,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-02 01:53:21,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-02 01:53:21,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-02 01:53:21,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-02 01:53:21,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-02 01:53:21,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-02 01:53:21,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-02 01:53:21,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-02 01:53:21,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-02 01:53:21,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-02 01:53:21,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-02 01:53:21,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-02 01:53:21,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-02 01:53:21,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-02 01:53:21,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-02 01:53:21,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-02 01:53:21,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-02 01:53:21,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-02 01:53:21,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-02 01:53:21,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-02 01:53:21,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-02 01:53:21,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-02 01:53:21,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-02 01:53:21,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-02 01:53:21,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-02 01:53:21,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-02 01:53:21,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-02 01:53:21,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-02 01:53:21,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-02 01:53:21,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-02 01:53:21,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-02 01:53:21,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-02 01:53:21,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-02 01:53:21,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-02 01:53:21,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-02 01:53:21,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-02 01:53:21,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-02 01:53:21,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-02 01:53:21,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-02 01:53:21,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-02 01:53:21,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-02 01:53:21,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-02 01:53:21,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-02 01:53:21,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-02 01:53:21,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-02 01:53:21,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-02 01:53:21,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-02 01:53:21,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-02 01:53:21,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-02 01:53:21,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-02 01:53:21,153 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0065610792988867375); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00908866414480969); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0065610792988867375); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-02 01:53:21,153 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-02 01:53:21,153 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.952040131078073E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0065610792988867375); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00908866414480969); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0065610792988867375); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-02 01:53:21,365 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 308 ms to try 5000 different iterations.  Found a solution that moves 256 regions; Going from a computed imbalance of 6.952040131078073E-4 to a new imbalance of 2.1820713145167584E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.028444444444444446, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-02 01:53:21,365 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1066561784:83 , srv1119597093:83 , srv1221134651:83 , srv179136220:83 , srv1990819571:83 , srv2052182277:83 , srv2119968664:83 , srv308849759:83 , srv530320249:83 , srv1147696000:84 , srv115380471:84 , srv1157743116:84 , srv1191045713:84 , srv1215252073:84 , srv1295056832:84 , srv1347443490:84 , srv1449537124:84 , srv1460341015:84 , srv1686061562:84 , srv1772938074:84 , srv1859361440:84 , srv1920506643:84 , srv1971933930:84 , srv1978183626:84 , srv1980909166:84 , srv216882382:84 , srv226301555:84 , srv268620265:84 , srv33396993:84 , srv498004756:84 , srv566809373:84 , srv60285109:84 , srv65282260:84 , srv656809320:84 , srv726426302:84 , srv744325085:84 , srv87435059:84 , srv879437859:84 , srv944414039:84 , srv980055994:84 , srv1000807179:85 , srv1027584769:85 , srv1100047204:85 , srv1111138267:85 , srv1150300292:85 , srv1171518357:85 , srv1204549625:85 , srv1213843800:85 , srv122150217:85 , srv1235851470:85 , srv1379759343:85 , srv1416481897:85 , srv1421846551:85 , srv1507845262:85 , srv1590999645:85 , srv160889163:85 , srv1625429285:85 , srv1634521368:85 , srv1658567210:85 , srv1721531087:85 , srv1833407564:85 , srv1836254134:85 , srv1863127024:85 , srv1912785164:85 , srv1983221321:85 , srv2048519113:85 , srv2087147108:85 , srv2110121532:85 , srv2144746849:85 , srv219880756:85 , srv224053702:85 , srv231382324:85 , srv235406041:85 , srv261107455:85 , srv274744506:85 , srv286997461:85 , srv364705010:85 , srv486683234:85 , srv535737880:85 , srv586302199:85 , srv594853229:85 , srv606687536:85 , srv672451572:85 , srv722887952:85 , srv792899054:85 , srv916528039:85 , srv955354334:85 , srv972558944:85 , srv985744895:85 , srv994836731:85 , srv1057188421:86 , srv1109782307:86 , srv1161077576:86 , srv1210661196:86 , srv1260666902:86 , srv190427988:86 , srv1994419406:86 , srv2059159000:86 , srv74859161:86 , srv111809942:625 }
2024-03-02 01:53:21,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-02 01:53:21,370 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1460341015=30, srv1109782307=5, srv1147696000=9, srv274744506=69, srv1066561784=3, srv122150217=21, srv1295056832=24, srv65282260=83, srv1171518357=14, srv1235851470=22, srv972558944=96, srv1421846551=28, srv1416481897=27, srv286997461=70, srv33396993=72, srv606687536=82, srv1379759343=26, srv586302199=79, srv2048519113=54, srv226301555=64, srv1191045713=15, srv160889163=33, srv985744895=98, srv530320249=76, srv726426302=87, srv216882382=61, srv60285109=81, srv1978183626=49, srv1150300292=10, srv1215252073=19, srv2144746849=60, srv1119597093=8, srv944414039=94, srv1658567210=36, srv566809373=78, srv1221134651=20, srv955354334=95, srv980055994=97, srv994836731=99, srv1772938074=39, srv224053702=63, srv364705010=73, srv1213843800=18, srv179136220=40, srv1971933930=48, srv308849759=71, srv87435059=91, srv1686061562=37, srv1920506643=47, srv879437859=92, srv1000807179=0, srv1210661196=17, srv1057188421=2, srv594853229=80, srv1994419406=53, srv535737880=77, srv2059159000=56, srv115380471=11, srv1347443490=25, srv1859361440=43, srv1204549625=16, srv190427988=45, srv1625429285=34, srv1161077576=13, srv916528039=93, srv1100047204=4, srv1634521368=35, srv1157743116=12, srv261107455=67, srv656809320=84, srv1912785164=46, srv722887952=86, srv672451572=85, srv111809942=7, srv744325085=88, srv1836254134=42, srv1833407564=41, srv1983221321=51, srv1027584769=1, srv74859161=89, srv486683234=74, srv1590999645=32, srv1980909166=50, srv231382324=65, srv1721531087=38, srv219880756=62, srv2087147108=57, srv1260666902=23, srv1449537124=29, srv1863127024=44, srv792899054=90, srv498004756=75, srv235406041=66, srv268620265=68, srv1111138267=6, srv2119968664=59, srv2110121532=58, srv2052182277=55, srv1507845262=31, srv1990819571=52} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-02 01:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-02 01:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-02 01:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-02 01:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-02 01:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-02 01:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-02 01:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-02 01:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-02 01:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-02 01:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-02 01:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-02 01:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-02 01:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-02 01:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-02 01:53:21,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-02 01:53:21,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-02 01:53:21,383 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-02 01:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-02 01:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-02 01:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-02 01:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-02 01:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-02 01:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-02 01:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-02 01:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-02 01:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-02 01:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-02 01:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-02 01:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-02 01:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-02 01:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-02 01:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-02 01:53:21,384 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-02 01:53:21,385 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-02 01:53:21,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-02 01:53:21,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-02 01:53:21,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-02 01:53:21,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-02 01:53:21,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-02 01:53:21,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-02 01:53:21,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-02 01:53:21,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-02 01:53:21,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-02 01:53:21,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-02 01:53:21,386 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-02 01:53:21,492 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=2.002816283222151E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0022222222222222222); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-02 01:53:21,493 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1066561784:83 , srv1119597093:83 , srv1221134651:83 , srv179136220:83 , srv1990819571:83 , srv2052182277:83 , srv2119968664:83 , srv308849759:83 , srv530320249:83 , srv1147696000:84 , srv115380471:84 , srv1157743116:84 , srv1191045713:84 , srv1215252073:84 , srv1295056832:84 , srv1347443490:84 , srv1449537124:84 , srv1460341015:84 , srv1686061562:84 , srv1772938074:84 , srv1859361440:84 , srv1920506643:84 , srv1971933930:84 , srv1978183626:84 , srv1980909166:84 , srv216882382:84 , srv226301555:84 , srv268620265:84 , srv33396993:84 , srv498004756:84 , srv566809373:84 , srv60285109:84 , srv65282260:84 , srv656809320:84 , srv726426302:84 , srv744325085:84 , srv87435059:84 , srv879437859:84 , srv944414039:84 , srv980055994:84 , srv1000807179:85 , srv1027584769:85 , srv1100047204:85 , srv1111138267:85 , srv1150300292:85 , srv1171518357:85 , srv1204549625:85 , srv1213843800:85 , srv122150217:85 , srv1235851470:85 , srv1379759343:85 , srv1416481897:85 , srv1421846551:85 , srv1507845262:85 , srv1590999645:85 , srv160889163:85 , srv1625429285:85 , srv1634521368:85 , srv1658567210:85 , srv1721531087:85 , srv1833407564:85 , srv1836254134:85 , srv1863127024:85 , srv1912785164:85 , srv1983221321:85 , srv2048519113:85 , srv2087147108:85 , srv2110121532:85 , srv2144746849:85 , srv219880756:85 , srv224053702:85 , srv231382324:85 , srv235406041:85 , srv261107455:85 , srv274744506:85 , srv286997461:85 , srv364705010:85 , srv486683234:85 , srv535737880:85 , srv586302199:85 , srv594853229:85 , srv606687536:85 , srv672451572:85 , srv722887952:85 , srv792899054:85 , srv916528039:85 , srv955354334:85 , srv972558944:85 , srv985744895:85 , srv994836731:85 , srv1057188421:86 , srv1109782307:86 , srv1161077576:86 , srv1210661196:86 , srv1260666902:86 , srv190427988:86 , srv1994419406:86 , srv2059159000:86 , srv74859161:86 , srv111809942:625 }
2024-03-02 01:53:21,524 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=294 (was 277) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=9478 (was 10269)
2024-03-02 01:53:21,533 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=294, ProcessCount=172, AvailableMemoryMB=9478
2024-03-02 01:53:21,534 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-02 01:53:21,534 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-03-02 01:53:21,542 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1015233587:84 , srv1016448696:84 , srv1038702393:84 , srv1054055863:84 , srv1080289635:84 , srv1117483392:84 , srv1210855832:84 , srv1272315713:84 , srv142926138:84 , srv1445477186:84 , srv1451993096:84 , srv1550609671:84 , srv156160129:84 , srv1652294574:84 , srv1684284756:84 , srv1784822597:84 , srv1983032852:84 , srv2029439151:84 , srv2055272588:84 , srv2070742360:84 , srv236666126:84 , srv272750608:84 , srv321675051:84 , srv512929327:84 , srv521217839:84 , srv592148354:84 , srv619674723:84 , srv675661403:84 , srv818318252:84 , srv1542469012:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-03-02 01:53:21,543 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-02 01:53:21,544 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1542469012=11, srv675661403=28, srv1054055863=3, srv2070742360=20, srv1210855832=6, srv2055272588=19, srv1451993096=10, srv1983032852=17, srv156160129=13, srv619674723=27, srv1652294574=14, srv2029439151=18, srv1015233587=0, srv321675051=23, srv1784822597=16, srv1080289635=4, srv1684284756=15, srv1272315713=7, srv1445477186=9, srv512929327=24, srv592148354=26, srv1550609671=12, srv272750608=22, srv1016448696=1, srv521217839=25, srv1117483392=5, srv142926138=8, srv1038702393=2, srv236666126=21, srv818318252=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-02 01:53:21,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-02 01:53:21,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-02 01:53:21,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-02 01:53:21,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-02 01:53:21,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-02 01:53:21,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-02 01:53:21,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-02 01:53:21,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-02 01:53:21,556 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-02 01:53:21,557 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-02 01:53:21,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-02 01:53:21,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-02 01:53:21,558 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-02 01:53:21,573 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1088945025956636, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-02 01:53:21,573 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912536166286227, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1088945025956636, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-03-02 01:55:21,561 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6205305 different iterations.  Found a solution that moves 1850 regions; Going from a computed imbalance of 0.9912536166286227 to a new imbalance of 4.317992290299789E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6851851851851852, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-02 01:55:21,567 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1015233587:90 , srv1016448696:90 , srv1038702393:90 , srv1054055863:90 , srv1080289635:90 , srv1117483392:90 , srv1210855832:90 , srv1272315713:90 , srv142926138:90 , srv1445477186:90 , srv1451993096:90 , srv1542469012:90 , srv1550609671:90 , srv156160129:90 , srv1652294574:90 , srv1684284756:90 , srv1784822597:90 , srv1983032852:90 , srv2029439151:90 , srv2055272588:90 , srv2070742360:90 , srv236666126:90 , srv272750608:90 , srv321675051:90 , srv512929327:90 , srv521217839:90 , srv592148354:90 , srv619674723:90 , srv675661403:90 , srv818318252:90 }
2024-03-02 01:55:21,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-02 01:55:21,568 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1542469012=11, srv675661403=28, srv1054055863=3, srv2070742360=20, srv1210855832=6, srv2055272588=19, srv1451993096=10, srv1983032852=17, srv156160129=13, srv619674723=27, srv1652294574=14, srv2029439151=18, srv1015233587=0, srv321675051=23, srv1784822597=16, srv1080289635=4, srv1684284756=15, srv1272315713=7, srv1445477186=9, srv512929327=24, srv592148354=26, srv1550609671=12, srv272750608=22, srv1016448696=1, srv521217839=25, srv1117483392=5, srv142926138=8, srv1038702393=2, srv236666126=21, srv818318252=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-02 01:55:21,572 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-02 01:55:21,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-02 01:55:21,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-02 01:55:21,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-02 01:55:21,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-02 01:55:21,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-02 01:55:21,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-02 01:55:21,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-02 01:55:21,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-02 01:55:21,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-02 01:55:21,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-02 01:55:21,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-02 01:55:21,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-02 01:55:21,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-02 01:55:21,573 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-02 01:55:21,585 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-02 01:55:21,585 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1015233587:90 , srv1016448696:90 , srv1038702393:90 , srv1054055863:90 , srv1080289635:90 , srv1117483392:90 , srv1210855832:90 , srv1272315713:90 , srv142926138:90 , srv1445477186:90 , srv1451993096:90 , srv1542469012:90 , srv1550609671:90 , srv156160129:90 , srv1652294574:90 , srv1684284756:90 , srv1784822597:90 , srv1983032852:90 , srv2029439151:90 , srv2055272588:90 , srv2070742360:90 , srv236666126:90 , srv272750608:90 , srv321675051:90 , srv512929327:90 , srv521217839:90 , srv592148354:90 , srv619674723:90 , srv675661403:90 , srv818318252:90 }
2024-03-02 01:55:21,599 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=244 (was 294), ProcessCount=172 (was 172), AvailableMemoryMB=8583 (was 9478)