2024-03-27 14:53:26,158 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-03-27 14:53:26,454 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-03-27 14:53:26,571 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-03-27 14:53:26,631 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-27 14:53:26,638 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-27 14:53:26,654 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=236, ProcessCount=169, AvailableMemoryMB=11518
2024-03-27 14:53:26,656 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-27 14:53:26,656 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-27 14:53:26,773 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv102052729:84 , srv1020772197:84 , srv1029196747:84 , srv1040633865:84 , srv1045552849:84 , srv1111872702:84 , srv1113937249:84 , srv11288988:84 , srv1150699670:84 , srv120008823:84 , srv1203114624:84 , srv120398664:84 , srv1210178740:84 , srv1225853154:84 , srv1236664646:84 , srv1237376786:84 , srv1240223456:84 , srv1251330946:84 , srv1325338614:84 , srv1429217679:84 , srv143947160:84 , srv1442913364:84 , srv1463493437:84 , srv1490847401:84 , srv1537702928:84 , srv153785170:84 , srv1545473450:84 , srv1554993071:84 , srv1559447894:84 , srv1563801257:84 , srv1567037104:84 , srv1584442490:84 , srv1585348335:84 , srv159644809:84 , srv1599570123:84 , srv1640467057:84 , srv1673850852:84 , srv168100564:84 , srv1686006400:84 , srv1697726535:84 , srv1725776555:84 , srv1744713338:84 , srv1753849486:84 , srv176500894:84 , srv1770634153:84 , srv1801163971:84 , srv180193993:84 , srv1805396296:84 , srv1854965780:84 , srv1868517353:84 , srv1871993592:84 , srv1881943674:84 , srv1895811123:84 , srv1903522805:84 , srv1945741711:84 , srv1949217409:84 , srv1955572218:84 , srv1955672291:84 , srv2015555791:84 , srv2020994927:84 , srv2029248708:84 , srv2081919906:84 , srv2082464799:84 , srv2087238816:84 , srv209172719:84 , srv2113674309:84 , srv2127334754:84 , srv2138681373:84 , srv222271270:84 , srv224881532:84 , srv226040205:84 , srv276837948:84 , srv294994258:84 , srv315822643:84 , srv34357615:84 , srv358243702:84 , srv400599082:84 , srv434145551:84 , srv460608050:84 , srv463925424:84 , srv486019521:84 , srv492951608:84 , srv496718952:84 , srv520199151:84 , srv537881927:84 , srv55208104:84 , srv554678585:84 , srv559554873:84 , srv681758625:84 , srv684365610:84 , srv79323642:84 , srv813614737:84 , srv835823168:84 , srv879010877:84 , srv929080436:84 , srv93455446:84 , srv93759675:84 , srv983224569:84 , srv991160802:84 , srv1685275793:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-03-27 14:53:26,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-27 14:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1640467057=35, srv2138681373=68, srv434145551=78, srv1545473450=26, srv1903522805=54, srv153785170=25, srv2029248708=61, srv520199151=84, srv1559447894=28, srv1725776555=41, srv1150699670=8, srv1113937249=6, srv2081919906=62, srv879010877=94, srv1045552849=4, srv79323642=91, srv983224569=98, srv554678585=87, srv2113674309=66, srv1770634153=45, srv1203114624=10, srv1868517353=50, srv463925424=80, srv1744713338=42, srv1240223456=16, srv224881532=70, srv559554873=88, srv1686006400=39, srv176500894=44, srv358243702=76, srv1949217409=56, srv1881943674=52, srv1463493437=22, srv93455446=96, srv1225853154=13, srv460608050=79, srv1854965780=49, srv537881927=85, srv276837948=72, srv1567037104=30, srv1537702928=24, srv143947160=20, srv1801163971=46, srv400599082=77, srv1805396296=48, srv496718952=83, srv684365610=90, srv991160802=99, srv120398664=11, srv813614737=92, srv120008823=9, srv1871993592=51, srv1955572218=57, srv2127334754=67, srv1599570123=34, srv1563801257=29, srv1429217679=19, srv1585348335=32, srv1442913364=21, srv929080436=95, srv1697726535=40, srv2087238816=64, srv34357615=75, srv486019521=81, srv159644809=33, srv1111872702=5, srv1945741711=55, srv1955672291=58, srv209172719=65, srv1685275793=38, srv226040205=71, srv1554993071=27, srv315822643=74, srv168100564=37, srv2015555791=59, srv11288988=7, srv492951608=82, srv2082464799=63, srv55208104=86, srv1040633865=3, srv1210178740=12, srv1236664646=14, srv1029196747=2, srv681758625=89, srv1237376786=15, srv1895811123=53, srv102052729=0, srv1490847401=23, srv1584442490=31, srv222271270=69, srv180193993=47, srv2020994927=60, srv294994258=73, srv1020772197=1, srv1251330946=17, srv1673850852=36, srv93759675=97, srv1753849486=43, srv835823168=93, srv1325338614=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-27 14:53:26,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-27 14:53:26,856 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-27 14:53:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-27 14:53:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-27 14:53:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-27 14:53:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-27 14:53:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-27 14:53:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-27 14:53:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-27 14:53:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-27 14:53:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-27 14:53:26,857 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-27 14:53:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-27 14:53:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-27 14:53:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-27 14:53:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-27 14:53:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-27 14:53:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-27 14:53:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-27 14:53:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-27 14:53:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-27 14:53:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-27 14:53:26,858 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-27 14:53:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-27 14:53:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-27 14:53:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-27 14:53:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-27 14:53:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-27 14:53:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-27 14:53:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-27 14:53:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-27 14:53:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-27 14:53:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-27 14:53:26,859 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-27 14:53:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-27 14:53:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-27 14:53:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-27 14:53:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-27 14:53:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-27 14:53:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-27 14:53:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-27 14:53:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-27 14:53:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-27 14:53:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-27 14:53:26,860 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-27 14:53:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-27 14:53:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-27 14:53:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-27 14:53:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-27 14:53:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-27 14:53:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-27 14:53:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-27 14:53:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-27 14:53:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-27 14:53:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-27 14:53:26,861 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-27 14:53:26,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-27 14:53:26,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-27 14:53:26,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-27 14:53:26,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-27 14:53:26,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-27 14:53:26,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-27 14:53:26,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-27 14:53:26,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-27 14:53:26,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-27 14:53:26,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-27 14:53:26,862 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-27 14:53:26,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-27 14:53:26,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-27 14:53:26,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-27 14:53:26,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-27 14:53:26,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-27 14:53:26,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-27 14:53:26,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-27 14:53:26,863 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-27 14:53:26,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-27 14:53:26,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-27 14:53:26,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-27 14:53:26,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-27 14:53:26,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-27 14:53:26,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-27 14:53:26,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-27 14:53:26,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-27 14:53:26,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-27 14:53:26,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-27 14:53:26,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-27 14:53:26,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-27 14:53:26,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-27 14:53:26,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-27 14:53:26,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-27 14:53:26,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-27 14:53:26,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-27 14:53:26,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-27 14:53:26,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-27 14:53:26,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-27 14:53:26,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-27 14:53:26,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-27 14:53:26,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-27 14:53:26,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-27 14:53:26,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-27 14:53:26,866 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-27 14:53:27,038 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 14:53:27,039 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-27 14:53:27,039 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-27 14:53:27,387 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 493 ms to try 5000 different iterations.  Found a solution that moves 4676 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18446140412744888. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04206895212338543, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0615512159794767, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5195555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04206895212338543, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1795, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24825, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 14:53:27,404 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1429217679:82 , srv1585348335:82 , srv2015555791:82 , srv2082464799:82 , srv400599082:82 , srv835823168:82 , srv120398664:83 , srv1237376786:83 , srv153785170:83 , srv276837948:83 , srv55208104:83 , srv93759675:83 , srv1111872702:84 , srv120008823:84 , srv1251330946:84 , srv1325338614:84 , srv1554993071:84 , srv159644809:84 , srv1673850852:84 , srv1945741711:84 , srv2138681373:84 , srv224881532:84 , srv226040205:84 , srv813614737:84 , srv929080436:84 , srv1020772197:85 , srv1113937249:85 , srv1537702928:85 , srv1545473450:85 , srv176500894:85 , srv1895811123:85 , srv1955672291:85 , srv2087238816:85 , srv209172719:85 , srv463925424:85 , srv486019521:85 , srv879010877:85 , srv1045552849:86 , srv11288988:86 , srv1150699670:86 , srv1490847401:86 , srv1563801257:86 , srv1686006400:86 , srv1744713338:86 , srv1801163971:86 , srv1805396296:86 , srv1871993592:86 , srv1881943674:86 , srv1949217409:86 , srv2020994927:86 , srv2029248708:86 , srv294994258:86 , srv358243702:86 , srv520199151:86 , srv554678585:86 , srv93455446:86 , srv983224569:86 , srv1040633865:87 , srv1210178740:87 , srv1225853154:87 , srv143947160:87 , srv1463493437:87 , srv168100564:87 , srv1753849486:87 , srv180193993:87 , srv1854965780:87 , srv1868517353:87 , srv2127334754:87 , srv460608050:87 , srv492951608:87 , srv496718952:87 , srv537881927:87 , srv559554873:87 , srv684365610:87 , srv991160802:87 , srv102052729:88 , srv1203114624:88 , srv1236664646:88 , srv1697726535:88 , srv2081919906:88 , srv2113674309:88 , srv222271270:88 , srv34357615:88 , srv681758625:88 , srv79323642:88 , srv1240223456:89 , srv1442913364:89 , srv1567037104:89 , srv1640467057:89 , srv1725776555:89 , srv434145551:89 , srv1029196747:90 , srv1599570123:90 , srv1903522805:90 , srv1955572218:90 , srv1559447894:91 , srv315822643:92 , srv1770634153:94 , srv1584442490:97 , srv1685275793:464 }
2024-03-27 14:53:27,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-27 14:53:27,408 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1640467057=35, srv2138681373=68, srv434145551=78, srv1545473450=26, srv1903522805=54, srv153785170=25, srv2029248708=61, srv520199151=84, srv1559447894=28, srv1725776555=41, srv1150699670=8, srv1113937249=6, srv2081919906=62, srv879010877=94, srv1045552849=4, srv79323642=91, srv983224569=98, srv554678585=87, srv2113674309=66, srv1770634153=45, srv1203114624=10, srv1868517353=50, srv463925424=80, srv1744713338=42, srv1240223456=16, srv224881532=70, srv559554873=88, srv1686006400=39, srv176500894=44, srv358243702=76, srv1949217409=56, srv1881943674=52, srv1463493437=22, srv93455446=96, srv1225853154=13, srv460608050=79, srv1854965780=49, srv537881927=85, srv276837948=72, srv1567037104=30, srv1537702928=24, srv143947160=20, srv1801163971=46, srv400599082=77, srv1805396296=48, srv496718952=83, srv684365610=90, srv991160802=99, srv120398664=11, srv813614737=92, srv120008823=9, srv1871993592=51, srv1955572218=57, srv2127334754=67, srv1599570123=34, srv1563801257=29, srv1429217679=19, srv1585348335=32, srv1442913364=21, srv929080436=95, srv1697726535=40, srv2087238816=64, srv34357615=75, srv486019521=81, srv159644809=33, srv1111872702=5, srv1945741711=55, srv1955672291=58, srv209172719=65, srv1685275793=38, srv226040205=71, srv1554993071=27, srv315822643=74, srv168100564=37, srv2015555791=59, srv11288988=7, srv492951608=82, srv2082464799=63, srv55208104=86, srv1040633865=3, srv1210178740=12, srv1236664646=14, srv1029196747=2, srv681758625=89, srv1237376786=15, srv1895811123=53, srv102052729=0, srv1490847401=23, srv1584442490=31, srv222271270=69, srv180193993=47, srv2020994927=60, srv294994258=73, srv1020772197=1, srv1251330946=17, srv1673850852=36, srv93759675=97, srv1753849486=43, srv835823168=93, srv1325338614=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-27 14:53:27,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-27 14:53:27,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-27 14:53:27,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-27 14:53:27,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-27 14:53:27,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-27 14:53:27,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-27 14:53:27,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-27 14:53:27,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-27 14:53:27,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-27 14:53:27,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-27 14:53:27,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-27 14:53:27,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-27 14:53:27,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-27 14:53:27,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-27 14:53:27,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-27 14:53:27,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-27 14:53:27,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-27 14:53:27,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-27 14:53:27,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-27 14:53:27,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-27 14:53:27,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-27 14:53:27,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-27 14:53:27,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-27 14:53:27,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-27 14:53:27,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-27 14:53:27,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-27 14:53:27,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-27 14:53:27,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-27 14:53:27,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-27 14:53:27,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-27 14:53:27,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-27 14:53:27,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-27 14:53:27,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-27 14:53:27,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-27 14:53:27,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-27 14:53:27,423 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-27 14:53:27,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-27 14:53:27,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-27 14:53:27,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-27 14:53:27,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-27 14:53:27,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-27 14:53:27,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-27 14:53:27,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-27 14:53:27,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-27 14:53:27,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-27 14:53:27,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-27 14:53:27,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-27 14:53:27,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-27 14:53:27,424 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-27 14:53:27,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-27 14:53:27,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-27 14:53:27,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-27 14:53:27,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-27 14:53:27,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-27 14:53:27,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-27 14:53:27,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-27 14:53:27,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-27 14:53:27,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-27 14:53:27,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-27 14:53:27,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-27 14:53:27,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-27 14:53:27,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-27 14:53:27,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-27 14:53:27,425 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-27 14:53:27,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-27 14:53:27,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-27 14:53:27,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-27 14:53:27,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-27 14:53:27,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-27 14:53:27,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-27 14:53:27,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-27 14:53:27,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-27 14:53:27,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-27 14:53:27,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-27 14:53:27,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-27 14:53:27,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-27 14:53:27,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-27 14:53:27,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-27 14:53:27,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-27 14:53:27,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-27 14:53:27,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-27 14:53:27,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-27 14:53:27,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-27 14:53:27,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-27 14:53:27,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-27 14:53:27,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-27 14:53:27,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-27 14:53:27,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-27 14:53:27,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-27 14:53:27,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-27 14:53:27,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-27 14:53:27,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-27 14:53:27,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-27 14:53:27,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-27 14:53:27,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-27 14:53:27,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-27 14:53:27,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-27 14:53:27,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-27 14:53:27,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-27 14:53:27,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-27 14:53:27,428 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-27 14:53:27,554 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04206895212338543, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0615512159794767, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04206895212338543, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1795, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24825, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 14:53:27,555 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-27 14:53:27,555 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18442866207563896, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04206895212338543, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0615512159794767, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04206895212338543, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1795, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24825, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-27 14:53:27,774 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 329 ms to try 5000 different iterations.  Found a solution that moves 2671 regions; Going from a computed imbalance of 0.18442866207563896 to a new imbalance of 0.01735403808435179. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024300155725331184); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038446254409428206, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29677777777777775, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024300155725331184); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0145); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 14:53:27,785 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1584442490:74 , srv1770634153:78 , srv315822643:79 , srv1029196747:80 , srv1725776555:80 , srv1903522805:80 , srv1955572218:80 , srv1442913364:82 , srv1559447894:82 , srv1567037104:82 , srv1599570123:82 , srv168100564:82 , srv434145551:82 , srv1563801257:83 , srv2020994927:83 , srv1040633865:84 , srv11288988:84 , srv1203114624:84 , srv1463493437:84 , srv1490847401:84 , srv1697726535:84 , srv180193993:84 , srv1949217409:84 , srv2081919906:84 , srv222271270:84 , srv294994258:84 , srv537881927:84 , srv79323642:84 , srv1240223456:85 , srv1640467057:85 , srv1744713338:85 , srv1801163971:85 , srv1854965780:85 , srv1881943674:85 , srv1955672291:85 , srv2113674309:85 , srv2127334754:85 , srv34357615:85 , srv358243702:85 , srv486019521:85 , srv681758625:85 , srv879010877:85 , srv102052729:86 , srv1045552849:86 , srv1111872702:86 , srv1225853154:86 , srv1673850852:86 , srv1753849486:86 , srv1871993592:86 , srv2029248708:86 , srv460608050:86 , srv463925424:86 , srv559554873:86 , srv684365610:86 , srv93455446:86 , srv983224569:86 , srv1020772197:87 , srv1113937249:87 , srv1150699670:87 , srv120008823:87 , srv1210178740:87 , srv1237376786:87 , srv1537702928:87 , srv1554993071:87 , srv1585348335:87 , srv159644809:87 , srv1686006400:87 , srv1868517353:87 , srv1895811123:87 , srv209172719:87 , srv492951608:87 , srv496718952:87 , srv520199151:87 , srv554678585:87 , srv835823168:87 , srv929080436:87 , srv93759675:87 , srv120398664:88 , srv1236664646:88 , srv1251330946:88 , srv1325338614:88 , srv143947160:88 , srv1545473450:88 , srv1805396296:88 , srv224881532:88 , srv276837948:88 , srv813614737:88 , srv991160802:88 , srv176500894:89 , srv2015555791:89 , srv2082464799:89 , srv2087238816:89 , srv2138681373:89 , srv226040205:89 , srv1429217679:90 , srv153785170:90 , srv1945741711:90 , srv400599082:90 , srv55208104:90 , srv1685275793:526 }
2024-03-27 14:53:27,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-27 14:53:27,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1640467057=35, srv2138681373=68, srv434145551=78, srv1545473450=26, srv1903522805=54, srv153785170=25, srv2029248708=61, srv520199151=84, srv1559447894=28, srv1725776555=41, srv1150699670=8, srv1113937249=6, srv2081919906=62, srv879010877=94, srv1045552849=4, srv79323642=91, srv983224569=98, srv554678585=87, srv2113674309=66, srv1770634153=45, srv1203114624=10, srv1868517353=50, srv463925424=80, srv1744713338=42, srv1240223456=16, srv224881532=70, srv559554873=88, srv1686006400=39, srv176500894=44, srv358243702=76, srv1949217409=56, srv1881943674=52, srv1463493437=22, srv93455446=96, srv1225853154=13, srv460608050=79, srv1854965780=49, srv537881927=85, srv276837948=72, srv1567037104=30, srv1537702928=24, srv143947160=20, srv1801163971=46, srv400599082=77, srv1805396296=48, srv496718952=83, srv684365610=90, srv991160802=99, srv120398664=11, srv813614737=92, srv120008823=9, srv1871993592=51, srv1955572218=57, srv2127334754=67, srv1599570123=34, srv1563801257=29, srv1429217679=19, srv1585348335=32, srv1442913364=21, srv929080436=95, srv1697726535=40, srv2087238816=64, srv34357615=75, srv486019521=81, srv159644809=33, srv1111872702=5, srv1945741711=55, srv1955672291=58, srv209172719=65, srv1685275793=38, srv226040205=71, srv1554993071=27, srv315822643=74, srv168100564=37, srv2015555791=59, srv11288988=7, srv492951608=82, srv2082464799=63, srv55208104=86, srv1040633865=3, srv1210178740=12, srv1236664646=14, srv1029196747=2, srv681758625=89, srv1237376786=15, srv1895811123=53, srv102052729=0, srv1490847401=23, srv1584442490=31, srv222271270=69, srv180193993=47, srv2020994927=60, srv294994258=73, srv1020772197=1, srv1251330946=17, srv1673850852=36, srv93759675=97, srv1753849486=43, srv835823168=93, srv1325338614=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-27 14:53:27,830 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-27 14:53:27,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-27 14:53:27,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-27 14:53:27,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-27 14:53:27,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-27 14:53:27,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-27 14:53:27,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-27 14:53:27,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-27 14:53:27,831 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-27 14:53:27,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-27 14:53:27,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-27 14:53:27,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-27 14:53:27,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-27 14:53:27,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-27 14:53:27,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-27 14:53:27,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-27 14:53:27,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-27 14:53:27,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-27 14:53:27,832 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-27 14:53:27,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-27 14:53:27,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-27 14:53:27,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-27 14:53:27,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-27 14:53:27,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-27 14:53:27,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-27 14:53:27,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-27 14:53:27,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-27 14:53:27,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-27 14:53:27,833 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-27 14:53:27,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-27 14:53:27,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-27 14:53:27,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-27 14:53:27,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-27 14:53:27,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-27 14:53:27,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-27 14:53:27,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-27 14:53:27,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-27 14:53:27,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-27 14:53:27,834 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-27 14:53:27,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-27 14:53:27,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-27 14:53:27,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-27 14:53:27,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-27 14:53:27,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-27 14:53:27,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-27 14:53:27,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-27 14:53:27,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-27 14:53:27,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-27 14:53:27,835 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-27 14:53:27,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-27 14:53:27,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-27 14:53:27,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-27 14:53:27,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-27 14:53:27,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-27 14:53:27,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-27 14:53:27,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-27 14:53:27,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-27 14:53:27,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-27 14:53:27,836 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-27 14:53:27,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-27 14:53:27,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-27 14:53:27,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-27 14:53:27,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-27 14:53:27,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-27 14:53:27,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-27 14:53:27,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-27 14:53:27,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-27 14:53:27,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-27 14:53:27,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-27 14:53:27,837 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-27 14:53:27,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-27 14:53:27,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-27 14:53:27,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-27 14:53:27,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-27 14:53:27,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-27 14:53:27,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-27 14:53:27,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-27 14:53:27,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-27 14:53:27,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-27 14:53:27,838 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-27 14:53:27,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-27 14:53:27,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-27 14:53:27,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-27 14:53:27,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-27 14:53:27,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-27 14:53:27,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-27 14:53:27,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-27 14:53:27,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-27 14:53:27,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-27 14:53:27,839 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-27 14:53:27,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-27 14:53:27,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-27 14:53:27,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-27 14:53:27,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-27 14:53:27,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-27 14:53:27,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-27 14:53:27,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-27 14:53:27,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-27 14:53:27,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-27 14:53:27,840 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-27 14:53:27,841 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-27 14:53:27,984 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024300155725331184); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038446254409428206, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024300155725331184); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0145); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 14:53:27,984 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-27 14:53:27,984 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.017335335342610076, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.024300155725331184); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.038446254409428206, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.024300155725331184); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0145); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.044333333333333336, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-27 14:53:28,275 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 409 ms to try 5000 different iterations.  Found a solution that moves 949 regions; Going from a computed imbalance of 0.017335335342610076 to a new imbalance of 9.202273483708078E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007336359610916769); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010106059344065263); MoveCostFunction : (multiplier=7.0, imbalance=0.10544444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007336359610916769); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 14:53:28,277 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1236664646:81 , srv143947160:82 , srv1868517353:82 , srv1945741711:83 , srv991160802:83 , srv102052729:84 , srv1210178740:84 , srv1225853154:84 , srv1240223456:84 , srv1251330946:84 , srv153785170:84 , srv1545473450:84 , srv1640467057:84 , srv1753849486:84 , srv176500894:84 , srv1805396296:84 , srv1895811123:84 , srv2081919906:84 , srv2087238816:84 , srv2138681373:84 , srv34357615:84 , srv460608050:84 , srv492951608:84 , srv496718952:84 , srv520199151:84 , srv554678585:84 , srv559554873:84 , srv681758625:84 , srv684365610:84 , srv79323642:84 , srv983224569:84 , srv1020772197:85 , srv1113937249:85 , srv1150699670:85 , srv1203114624:85 , srv1325338614:85 , srv1559447894:85 , srv1599570123:85 , srv1686006400:85 , srv1697726535:85 , srv1770634153:85 , srv1854965780:85 , srv1871993592:85 , srv209172719:85 , srv2113674309:85 , srv224881532:85 , srv226040205:85 , srv400599082:85 , srv537881927:85 , srv813614737:85 , srv93455446:85 , srv1045552849:86 , srv120008823:86 , srv120398664:86 , srv1429217679:86 , srv1442913364:86 , srv1463493437:86 , srv1567037104:86 , srv1584442490:86 , srv159644809:86 , srv1744713338:86 , srv1801163971:86 , srv1881943674:86 , srv1955672291:86 , srv2015555791:86 , srv2029248708:86 , srv2127334754:86 , srv222271270:86 , srv55208104:86 , srv929080436:86 , srv1029196747:87 , srv1040633865:87 , srv1111872702:87 , srv1237376786:87 , srv1490847401:87 , srv1537702928:87 , srv1554993071:87 , srv1585348335:87 , srv180193993:87 , srv1903522805:87 , srv1955572218:87 , srv2082464799:87 , srv276837948:87 , srv294994258:87 , srv315822643:87 , srv358243702:87 , srv434145551:87 , srv463925424:87 , srv11288988:88 , srv1563801257:88 , srv1673850852:88 , srv1725776555:88 , srv1949217409:88 , srv486019521:88 , srv835823168:88 , srv879010877:88 , srv168100564:89 , srv93759675:89 , srv2020994927:90 , srv1685275793:533 }
2024-03-27 14:53:28,280 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-27 14:53:28,281 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1640467057=35, srv2138681373=68, srv434145551=78, srv1545473450=26, srv1903522805=54, srv153785170=25, srv2029248708=61, srv520199151=84, srv1559447894=28, srv1725776555=41, srv1150699670=8, srv1113937249=6, srv2081919906=62, srv879010877=94, srv1045552849=4, srv79323642=91, srv983224569=98, srv554678585=87, srv2113674309=66, srv1770634153=45, srv1203114624=10, srv1868517353=50, srv463925424=80, srv1744713338=42, srv1240223456=16, srv224881532=70, srv559554873=88, srv1686006400=39, srv176500894=44, srv358243702=76, srv1949217409=56, srv1881943674=52, srv1463493437=22, srv93455446=96, srv1225853154=13, srv460608050=79, srv1854965780=49, srv537881927=85, srv276837948=72, srv1567037104=30, srv1537702928=24, srv143947160=20, srv1801163971=46, srv400599082=77, srv1805396296=48, srv496718952=83, srv684365610=90, srv991160802=99, srv120398664=11, srv813614737=92, srv120008823=9, srv1871993592=51, srv1955572218=57, srv2127334754=67, srv1599570123=34, srv1563801257=29, srv1429217679=19, srv1585348335=32, srv1442913364=21, srv929080436=95, srv1697726535=40, srv2087238816=64, srv34357615=75, srv486019521=81, srv159644809=33, srv1111872702=5, srv1945741711=55, srv1955672291=58, srv209172719=65, srv1685275793=38, srv226040205=71, srv1554993071=27, srv315822643=74, srv168100564=37, srv2015555791=59, srv11288988=7, srv492951608=82, srv2082464799=63, srv55208104=86, srv1040633865=3, srv1210178740=12, srv1236664646=14, srv1029196747=2, srv681758625=89, srv1237376786=15, srv1895811123=53, srv102052729=0, srv1490847401=23, srv1584442490=31, srv222271270=69, srv180193993=47, srv2020994927=60, srv294994258=73, srv1020772197=1, srv1251330946=17, srv1673850852=36, srv93759675=97, srv1753849486=43, srv835823168=93, srv1325338614=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-27 14:53:28,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-27 14:53:28,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-27 14:53:28,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-27 14:53:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-27 14:53:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-27 14:53:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-27 14:53:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-27 14:53:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-27 14:53:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-27 14:53:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-27 14:53:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-27 14:53:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-27 14:53:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-27 14:53:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-27 14:53:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-27 14:53:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-27 14:53:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-27 14:53:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-27 14:53:28,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-27 14:53:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-27 14:53:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-27 14:53:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-27 14:53:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-27 14:53:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-27 14:53:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-27 14:53:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-27 14:53:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-27 14:53:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-27 14:53:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-27 14:53:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-27 14:53:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-27 14:53:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-27 14:53:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-27 14:53:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-27 14:53:28,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-27 14:53:28,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-27 14:53:28,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-27 14:53:28,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-27 14:53:28,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-27 14:53:28,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-27 14:53:28,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-27 14:53:28,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-27 14:53:28,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-27 14:53:28,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-27 14:53:28,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-27 14:53:28,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-27 14:53:28,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-27 14:53:28,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-27 14:53:28,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-27 14:53:28,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-27 14:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-27 14:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-27 14:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-27 14:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-27 14:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-27 14:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-27 14:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-27 14:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-27 14:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-27 14:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-27 14:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-27 14:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-27 14:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-27 14:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-27 14:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-27 14:53:28,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-27 14:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-27 14:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-27 14:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-27 14:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-27 14:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-27 14:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-27 14:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-27 14:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-27 14:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-27 14:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-27 14:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-27 14:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-27 14:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-27 14:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-27 14:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-27 14:53:28,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-27 14:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-27 14:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-27 14:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-27 14:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-27 14:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-27 14:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-27 14:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-27 14:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-27 14:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-27 14:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-27 14:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-27 14:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-27 14:53:28,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-27 14:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-27 14:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-27 14:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-27 14:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-27 14:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-27 14:53:28,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-27 14:53:28,416 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007336359610916769); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010106059344065263); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007336359610916769); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 14:53:28,417 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-27 14:53:28,417 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=9.135823083435194E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007336359610916769); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010106059344065263); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.007336359610916769); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-27 14:53:28,770 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 460 ms to try 5000 different iterations.  Found a solution that moves 286 regions; Going from a computed imbalance of 9.135823083435194E-4 to a new imbalance of 5.791116487302144E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.03177777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 14:53:28,772 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2020994927:83 , srv2127334754:83 , srv315822643:83 , srv55208104:83 , srv93759675:83 , srv102052729:84 , srv1040633865:84 , srv1045552849:84 , srv1113937249:84 , srv11288988:84 , srv1150699670:84 , srv1210178740:84 , srv1236664646:84 , srv1463493437:84 , srv1537702928:84 , srv1554993071:84 , srv1559447894:84 , srv1567037104:84 , srv1673850852:84 , srv168100564:84 , srv1686006400:84 , srv1744713338:84 , srv180193993:84 , srv1805396296:84 , srv1903522805:84 , srv1949217409:84 , srv2029248708:84 , srv2082464799:84 , srv2113674309:84 , srv224881532:84 , srv276837948:84 , srv34357615:84 , srv358243702:84 , srv434145551:84 , srv486019521:84 , srv492951608:84 , srv496718952:84 , srv684365610:84 , srv879010877:84 , srv1020772197:85 , srv1029196747:85 , srv1111872702:85 , srv120008823:85 , srv1203114624:85 , srv120398664:85 , srv1225853154:85 , srv1237376786:85 , srv1240223456:85 , srv1325338614:85 , srv1429217679:85 , srv1442913364:85 , srv1490847401:85 , srv153785170:85 , srv1545473450:85 , srv1563801257:85 , srv1584442490:85 , srv159644809:85 , srv1599570123:85 , srv1640467057:85 , srv1697726535:85 , srv1725776555:85 , srv1753849486:85 , srv176500894:85 , srv1770634153:85 , srv1801163971:85 , srv1854965780:85 , srv1871993592:85 , srv1881943674:85 , srv1945741711:85 , srv1955572218:85 , srv2015555791:85 , srv2087238816:85 , srv209172719:85 , srv2138681373:85 , srv222271270:85 , srv226040205:85 , srv294994258:85 , srv400599082:85 , srv460608050:85 , srv463925424:85 , srv520199151:85 , srv554678585:85 , srv559554873:85 , srv681758625:85 , srv813614737:85 , srv835823168:85 , srv929080436:85 , srv93455446:85 , srv983224569:85 , srv991160802:85 , srv1251330946:86 , srv143947160:86 , srv1585348335:86 , srv1868517353:86 , srv1895811123:86 , srv1955672291:86 , srv2081919906:86 , srv79323642:86 , srv537881927:87 , srv1685275793:619 }
2024-03-27 14:53:28,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-27 14:53:28,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1640467057=35, srv2138681373=68, srv434145551=78, srv1545473450=26, srv1903522805=54, srv153785170=25, srv2029248708=61, srv520199151=84, srv1559447894=28, srv1725776555=41, srv1150699670=8, srv1113937249=6, srv2081919906=62, srv879010877=94, srv1045552849=4, srv79323642=91, srv983224569=98, srv554678585=87, srv2113674309=66, srv1770634153=45, srv1203114624=10, srv1868517353=50, srv463925424=80, srv1744713338=42, srv1240223456=16, srv224881532=70, srv559554873=88, srv1686006400=39, srv176500894=44, srv358243702=76, srv1949217409=56, srv1881943674=52, srv1463493437=22, srv93455446=96, srv1225853154=13, srv460608050=79, srv1854965780=49, srv537881927=85, srv276837948=72, srv1567037104=30, srv1537702928=24, srv143947160=20, srv1801163971=46, srv400599082=77, srv1805396296=48, srv496718952=83, srv684365610=90, srv991160802=99, srv120398664=11, srv813614737=92, srv120008823=9, srv1871993592=51, srv1955572218=57, srv2127334754=67, srv1599570123=34, srv1563801257=29, srv1429217679=19, srv1585348335=32, srv1442913364=21, srv929080436=95, srv1697726535=40, srv2087238816=64, srv34357615=75, srv486019521=81, srv159644809=33, srv1111872702=5, srv1945741711=55, srv1955672291=58, srv209172719=65, srv1685275793=38, srv226040205=71, srv1554993071=27, srv315822643=74, srv168100564=37, srv2015555791=59, srv11288988=7, srv492951608=82, srv2082464799=63, srv55208104=86, srv1040633865=3, srv1210178740=12, srv1236664646=14, srv1029196747=2, srv681758625=89, srv1237376786=15, srv1895811123=53, srv102052729=0, srv1490847401=23, srv1584442490=31, srv222271270=69, srv180193993=47, srv2020994927=60, srv294994258=73, srv1020772197=1, srv1251330946=17, srv1673850852=36, srv93759675=97, srv1753849486=43, srv835823168=93, srv1325338614=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-27 14:53:28,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-27 14:53:28,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-27 14:53:28,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-27 14:53:28,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-27 14:53:28,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-27 14:53:28,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-27 14:53:28,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-27 14:53:28,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-27 14:53:28,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-27 14:53:28,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-27 14:53:28,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-27 14:53:28,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-27 14:53:28,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-27 14:53:28,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-27 14:53:28,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-27 14:53:28,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-27 14:53:28,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-27 14:53:28,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-27 14:53:28,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-27 14:53:28,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-27 14:53:28,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-27 14:53:28,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-27 14:53:28,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-27 14:53:28,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-27 14:53:28,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-27 14:53:28,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-27 14:53:28,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-27 14:53:28,795 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-27 14:53:28,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-27 14:53:28,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-27 14:53:28,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-27 14:53:28,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-27 14:53:28,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-27 14:53:28,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-27 14:53:28,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-27 14:53:28,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-27 14:53:28,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-27 14:53:28,796 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-27 14:53:28,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-27 14:53:28,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-27 14:53:28,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-27 14:53:28,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-27 14:53:28,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-27 14:53:28,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-27 14:53:28,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-27 14:53:28,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-27 14:53:28,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-27 14:53:28,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-27 14:53:28,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-27 14:53:28,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-27 14:53:28,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-27 14:53:28,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-27 14:53:28,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-27 14:53:28,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-27 14:53:28,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-27 14:53:28,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-27 14:53:28,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-27 14:53:28,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-27 14:53:28,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-27 14:53:28,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-27 14:53:28,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-27 14:53:28,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-27 14:53:28,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-27 14:53:28,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-27 14:53:28,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-27 14:53:28,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-27 14:53:28,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-27 14:53:28,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-27 14:53:28,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-27 14:53:28,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-27 14:53:28,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-27 14:53:28,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-27 14:53:28,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-27 14:53:28,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-27 14:53:28,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-27 14:53:28,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-27 14:53:28,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-27 14:53:28,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-27 14:53:28,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-27 14:53:28,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-27 14:53:28,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-27 14:53:28,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-27 14:53:28,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-27 14:53:28,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-27 14:53:28,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-27 14:53:28,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-27 14:53:28,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-27 14:53:28,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-27 14:53:28,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-27 14:53:28,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-27 14:53:28,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-27 14:53:28,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-27 14:53:28,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-27 14:53:28,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-27 14:53:28,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-27 14:53:28,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-27 14:53:28,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-27 14:53:28,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-27 14:53:28,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-27 14:53:28,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-27 14:53:28,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-27 14:53:28,916 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 14:53:28,916 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-27 14:53:28,917 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=5.5908550070277E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.470732932414788E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.470732932414788E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=5.0E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-27 14:53:29,157 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 343 ms to try 5000 different iterations.  Found a solution that moves 37 regions; Going from a computed imbalance of 5.5908550070277E-5 to a new imbalance of 6.657065209687686E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.004111111111111111); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 14:53:29,158 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1240223456:83 , srv1429217679:83 , srv143947160:83 , srv1537702928:83 , srv1640467057:83 , srv176500894:83 , srv2087238816:83 , srv222271270:83 , srv226040205:83 , srv463925424:83 , srv537881927:83 , srv991160802:83 , srv102052729:84 , srv1020772197:84 , srv1029196747:84 , srv1040633865:84 , srv1045552849:84 , srv1111872702:84 , srv11288988:84 , srv1150699670:84 , srv120008823:84 , srv1203114624:84 , srv120398664:84 , srv1210178740:84 , srv1225853154:84 , srv1237376786:84 , srv1251330946:84 , srv1325338614:84 , srv1442913364:84 , srv1490847401:84 , srv153785170:84 , srv1545473450:84 , srv1554993071:84 , srv1559447894:84 , srv1563801257:84 , srv1584442490:84 , srv1585348335:84 , srv159644809:84 , srv1599570123:84 , srv1673850852:84 , srv168100564:84 , srv1686006400:84 , srv1697726535:84 , srv1725776555:84 , srv1753849486:84 , srv1770634153:84 , srv1801163971:84 , srv180193993:84 , srv1805396296:84 , srv1854965780:84 , srv1868517353:84 , srv1871993592:84 , srv1881943674:84 , srv1895811123:84 , srv1945741711:84 , srv1949217409:84 , srv1955572218:84 , srv1955672291:84 , srv2015555791:84 , srv2020994927:84 , srv2029248708:84 , srv2081919906:84 , srv2082464799:84 , srv209172719:84 , srv2113674309:84 , srv2138681373:84 , srv276837948:84 , srv294994258:84 , srv358243702:84 , srv400599082:84 , srv434145551:84 , srv460608050:84 , srv486019521:84 , srv492951608:84 , srv496718952:84 , srv520199151:84 , srv55208104:84 , srv554678585:84 , srv559554873:84 , srv681758625:84 , srv79323642:84 , srv813614737:84 , srv835823168:84 , srv879010877:84 , srv929080436:84 , srv93455446:84 , srv93759675:84 , srv1113937249:85 , srv1236664646:85 , srv1463493437:85 , srv1567037104:85 , srv1744713338:85 , srv1903522805:85 , srv2127334754:85 , srv224881532:85 , srv315822643:85 , srv34357615:85 , srv684365610:85 , srv983224569:85 , srv1685275793:684 }
2024-03-27 14:53:29,161 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-27 14:53:29,162 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1640467057=35, srv2138681373=68, srv434145551=78, srv1545473450=26, srv1903522805=54, srv153785170=25, srv2029248708=61, srv520199151=84, srv1559447894=28, srv1725776555=41, srv1150699670=8, srv1113937249=6, srv2081919906=62, srv879010877=94, srv1045552849=4, srv79323642=91, srv983224569=98, srv554678585=87, srv2113674309=66, srv1770634153=45, srv1203114624=10, srv1868517353=50, srv463925424=80, srv1744713338=42, srv1240223456=16, srv224881532=70, srv559554873=88, srv1686006400=39, srv176500894=44, srv358243702=76, srv1949217409=56, srv1881943674=52, srv1463493437=22, srv93455446=96, srv1225853154=13, srv460608050=79, srv1854965780=49, srv537881927=85, srv276837948=72, srv1567037104=30, srv1537702928=24, srv143947160=20, srv1801163971=46, srv400599082=77, srv1805396296=48, srv496718952=83, srv684365610=90, srv991160802=99, srv120398664=11, srv813614737=92, srv120008823=9, srv1871993592=51, srv1955572218=57, srv2127334754=67, srv1599570123=34, srv1563801257=29, srv1429217679=19, srv1585348335=32, srv1442913364=21, srv929080436=95, srv1697726535=40, srv2087238816=64, srv34357615=75, srv486019521=81, srv159644809=33, srv1111872702=5, srv1945741711=55, srv1955672291=58, srv209172719=65, srv1685275793=38, srv226040205=71, srv1554993071=27, srv315822643=74, srv168100564=37, srv2015555791=59, srv11288988=7, srv492951608=82, srv2082464799=63, srv55208104=86, srv1040633865=3, srv1210178740=12, srv1236664646=14, srv1029196747=2, srv681758625=89, srv1237376786=15, srv1895811123=53, srv102052729=0, srv1490847401=23, srv1584442490=31, srv222271270=69, srv180193993=47, srv2020994927=60, srv294994258=73, srv1020772197=1, srv1251330946=17, srv1673850852=36, srv93759675=97, srv1753849486=43, srv835823168=93, srv1325338614=18} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-27 14:53:29,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-27 14:53:29,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-27 14:53:29,179 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-27 14:53:29,180 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-27 14:53:29,181 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-27 14:53:29,182 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-27 14:53:29,183 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-27 14:53:29,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-27 14:53:29,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-27 14:53:29,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-27 14:53:29,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-27 14:53:29,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-27 14:53:29,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-27 14:53:29,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-27 14:53:29,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-27 14:53:29,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-27 14:53:29,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-27 14:53:29,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-27 14:53:29,184 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-27 14:53:29,297 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 14:53:29,298 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1240223456:83 , srv1429217679:83 , srv143947160:83 , srv1537702928:83 , srv1640467057:83 , srv176500894:83 , srv2087238816:83 , srv222271270:83 , srv226040205:83 , srv463925424:83 , srv537881927:83 , srv991160802:83 , srv102052729:84 , srv1020772197:84 , srv1029196747:84 , srv1040633865:84 , srv1045552849:84 , srv1111872702:84 , srv11288988:84 , srv1150699670:84 , srv120008823:84 , srv1203114624:84 , srv120398664:84 , srv1210178740:84 , srv1225853154:84 , srv1237376786:84 , srv1251330946:84 , srv1325338614:84 , srv1442913364:84 , srv1490847401:84 , srv153785170:84 , srv1545473450:84 , srv1554993071:84 , srv1559447894:84 , srv1563801257:84 , srv1584442490:84 , srv1585348335:84 , srv159644809:84 , srv1599570123:84 , srv1673850852:84 , srv168100564:84 , srv1686006400:84 , srv1697726535:84 , srv1725776555:84 , srv1753849486:84 , srv1770634153:84 , srv1801163971:84 , srv180193993:84 , srv1805396296:84 , srv1854965780:84 , srv1868517353:84 , srv1871993592:84 , srv1881943674:84 , srv1895811123:84 , srv1945741711:84 , srv1949217409:84 , srv1955572218:84 , srv1955672291:84 , srv2015555791:84 , srv2020994927:84 , srv2029248708:84 , srv2081919906:84 , srv2082464799:84 , srv209172719:84 , srv2113674309:84 , srv2138681373:84 , srv276837948:84 , srv294994258:84 , srv358243702:84 , srv400599082:84 , srv434145551:84 , srv460608050:84 , srv486019521:84 , srv492951608:84 , srv496718952:84 , srv520199151:84 , srv55208104:84 , srv554678585:84 , srv559554873:84 , srv681758625:84 , srv79323642:84 , srv813614737:84 , srv835823168:84 , srv879010877:84 , srv929080436:84 , srv93455446:84 , srv93759675:84 , srv1113937249:85 , srv1236664646:85 , srv1463493437:85 , srv1567037104:85 , srv1744713338:85 , srv1903522805:85 , srv2127334754:85 , srv224881532:85 , srv315822643:85 , srv34357615:85 , srv684365610:85 , srv983224569:85 , srv1685275793:684 }
2024-03-27 14:53:29,325 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=249 (was 236) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=11015 (was 11518)
2024-03-27 14:53:29,334 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=249, ProcessCount=169, AvailableMemoryMB=11014
2024-03-27 14:53:29,335 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-27 14:53:29,335 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-03-27 14:53:29,346 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv105748617:84 , srv1087285797:84 , srv122621756:84 , srv123163259:84 , srv1328430499:84 , srv1353967375:84 , srv1530293443:84 , srv1575439574:84 , srv1700227926:84 , srv1764188376:84 , srv1839498230:84 , srv1890190874:84 , srv1912022481:84 , srv1935914904:84 , srv1961823584:84 , srv2052501446:84 , srv2087227124:84 , srv208795386:84 , srv2109508154:84 , srv2129154823:84 , srv545294520:84 , srv632765834:84 , srv788549715:84 , srv835058985:84 , srv871637270:84 , srv906458461:84 , srv923755462:84 , srv992365785:84 , srv996425155:84 , srv1080275096:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-03-27 14:53:29,348 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-27 14:53:29,349 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1912022481=13, srv2052501446=16, srv1328430499=5, srv1890190874=12, srv2109508154=19, srv632765834=22, srv992365785=28, srv1961823584=15, srv2129154823=20, srv122621756=3, srv871637270=25, srv123163259=4, srv1935914904=14, srv1839498230=11, srv2087227124=17, srv1087285797=2, srv906458461=26, srv1764188376=10, srv208795386=18, srv1353967375=6, srv105748617=0, srv835058985=24, srv923755462=27, srv1080275096=1, srv1530293443=7, srv1700227926=9, srv1575439574=8, srv545294520=21, srv788549715=23, srv996425155=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-27 14:53:29,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-27 14:53:29,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-27 14:53:29,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-27 14:53:29,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-27 14:53:29,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-27 14:53:29,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-27 14:53:29,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-27 14:53:29,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-27 14:53:29,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-27 14:53:29,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-27 14:53:29,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-27 14:53:29,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-27 14:53:29,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-27 14:53:29,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-27 14:53:29,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-27 14:53:29,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-27 14:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-27 14:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-27 14:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-27 14:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-27 14:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-27 14:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-27 14:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-27 14:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-27 14:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-27 14:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-27 14:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-27 14:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-27 14:53:29,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-27 14:53:29,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-27 14:53:29,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-27 14:53:29,395 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0822022336958874, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 14:53:29,396 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912452059818507, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0822022336958874, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-03-27 14:55:29,369 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5583280 different iterations.  Found a solution that moves 1866 regions; Going from a computed imbalance of 0.9912452059818507 to a new imbalance of 4.3553370884861656E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6911111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 14:55:29,376 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv105748617:90 , srv1080275096:90 , srv1087285797:90 , srv122621756:90 , srv123163259:90 , srv1328430499:90 , srv1353967375:90 , srv1530293443:90 , srv1575439574:90 , srv1700227926:90 , srv1764188376:90 , srv1839498230:90 , srv1890190874:90 , srv1912022481:90 , srv1935914904:90 , srv1961823584:90 , srv2052501446:90 , srv2087227124:90 , srv208795386:90 , srv2109508154:90 , srv2129154823:90 , srv545294520:90 , srv632765834:90 , srv788549715:90 , srv835058985:90 , srv871637270:90 , srv906458461:90 , srv923755462:90 , srv992365785:90 , srv996425155:90 }
2024-03-27 14:55:29,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-27 14:55:29,377 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1912022481=13, srv2052501446=16, srv1328430499=5, srv1890190874=12, srv2109508154=19, srv632765834=22, srv992365785=28, srv1961823584=15, srv2129154823=20, srv122621756=3, srv871637270=25, srv123163259=4, srv1935914904=14, srv1839498230=11, srv2087227124=17, srv1087285797=2, srv906458461=26, srv1764188376=10, srv208795386=18, srv1353967375=6, srv105748617=0, srv835058985=24, srv923755462=27, srv1080275096=1, srv1530293443=7, srv1700227926=9, srv1575439574=8, srv545294520=21, srv788549715=23, srv996425155=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-27 14:55:29,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-27 14:55:29,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-27 14:55:29,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-27 14:55:29,380 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-27 14:55:29,381 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-27 14:55:29,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-27 14:55:29,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-27 14:55:29,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-27 14:55:29,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-27 14:55:29,382 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-27 14:55:29,395 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-27 14:55:29,395 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv105748617:90 , srv1080275096:90 , srv1087285797:90 , srv122621756:90 , srv123163259:90 , srv1328430499:90 , srv1353967375:90 , srv1530293443:90 , srv1575439574:90 , srv1700227926:90 , srv1764188376:90 , srv1839498230:90 , srv1890190874:90 , srv1912022481:90 , srv1935914904:90 , srv1961823584:90 , srv2052501446:90 , srv2087227124:90 , srv208795386:90 , srv2109508154:90 , srv2129154823:90 , srv545294520:90 , srv632765834:90 , srv788549715:90 , srv835058985:90 , srv871637270:90 , srv906458461:90 , srv923755462:90 , srv992365785:90 , srv996425155:90 }
2024-03-27 14:55:29,409 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=165 (was 249), ProcessCount=168 (was 169), AvailableMemoryMB=10242 (was 11014)