2024-02-24 12:53:14,970 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-02-24 12:53:15,321 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-02-24 12:53:15,447 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-02-24 12:53:15,508 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-24 12:53:15,516 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-02-24 12:53:15,532 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=248, ProcessCount=171, AvailableMemoryMB=21945
2024-02-24 12:53:15,535 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-24 12:53:15,535 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-02-24 12:53:15,674 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv101571443:84 , srv1029394172:84 , srv1030176332:84 , srv1036582822:84 , srv106242799:84 , srv1090722484:84 , srv1090853466:84 , srv1097346302:84 , srv110243249:84 , srv1138099888:84 , srv1138826844:84 , srv1145661967:84 , srv1190469892:84 , srv1218453043:84 , srv1321539842:84 , srv1369271437:84 , srv1374195671:84 , srv1377920783:84 , srv1380562253:84 , srv1386594810:84 , srv139304918:84 , srv1396845667:84 , srv1452993596:84 , srv1453974148:84 , srv1496155809:84 , srv1511666074:84 , srv1522678840:84 , srv1529274003:84 , srv1530162986:84 , srv1531444564:84 , srv1579442441:84 , srv1609017344:84 , srv165296794:84 , srv1669753075:84 , srv1672534903:84 , srv1680607619:84 , srv1686040548:84 , srv1689410608:84 , srv1739917469:84 , srv1752484082:84 , srv1769524880:84 , srv1799378523:84 , srv190868906:84 , srv1922552583:84 , srv1962139844:84 , srv1967725111:84 , srv1989789908:84 , srv1992834715:84 , srv1999921446:84 , srv2005419637:84 , srv2022731423:84 , srv2023859134:84 , srv2036130442:84 , srv2036971151:84 , srv2042425784:84 , srv2099603350:84 , srv2112867828:84 , srv2114130530:84 , srv2117043703:84 , srv2140567783:84 , srv218718837:84 , srv226031790:84 , srv280005225:84 , srv281638689:84 , srv307973174:84 , srv308359903:84 , srv332183356:84 , srv333771445:84 , srv341954614:84 , srv344945185:84 , srv363814586:84 , srv382249349:84 , srv405424632:84 , srv407757544:84 , srv416385781:84 , srv439298455:84 , srv481893161:84 , srv483361315:84 , srv488173331:84 , srv521076890:84 , srv618859188:84 , srv633271515:84 , srv640162272:84 , srv64428907:84 , srv655602838:84 , srv660641825:84 , srv726127470:84 , srv729744960:84 , srv7802002:84 , srv787981421:84 , srv789886155:84 , srv798514929:84 , srv881671704:84 , srv924635679:84 , srv935272039:84 , srv947235495:84 , srv965351614:84 , srv96585853:84 , srv975769766:84 , srv502353201:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-02-24 12:53:15,687 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-24 12:53:15,696 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1090722484=5, srv2005419637=49, srv947235495=96, srv363814586=70, srv344945185=69, srv1138099888=9, srv2023859134=51, srv1377920783=17, srv1989789908=46, srv1029394172=1, srv483361315=77, srv1967725111=45, srv1992834715=47, srv1369271437=15, srv341954614=68, srv281638689=63, srv2036971151=53, srv139304918=20, srv1669753075=33, srv1452993596=22, srv2112867828=56, srv190868906=42, srv502353201=79, srv2036130442=52, srv521076890=80, srv1769524880=40, srv881671704=93, srv1672534903=34, srv726127470=87, srv1380562253=18, srv382249349=71, srv407757544=73, srv280005225=62, srv332183356=66, srv935272039=95, srv1453974148=23, srv618859188=81, srv1680607619=35, srv308359903=65, srv1496155809=24, srv1799378523=41, srv1530162986=28, srv439298455=75, srv660641825=86, srv2022731423=50, srv110243249=8, srv787981421=90, srv1531444564=29, srv1097346302=7, srv789886155=91, srv975769766=99, srv1522678840=26, srv1190469892=12, srv1386594810=19, srv307973174=64, srv1609017344=31, srv1579442441=30, srv333771445=67, srv1138826844=10, srv633271515=82, srv1752484082=39, srv405424632=72, srv2099603350=55, srv1396845667=21, srv106242799=4, srv1686040548=36, srv226031790=61, srv1321539842=14, srv1511666074=25, srv165296794=32, srv64428907=84, srv1090853466=6, srv2114130530=57, srv101571443=0, srv1922552583=43, srv96585853=98, srv1962139844=44, srv218718837=60, srv655602838=85, srv488173331=78, srv1689410608=37, srv729744960=88, srv2117043703=58, srv416385781=74, srv1374195671=16, srv798514929=92, srv2140567783=59, srv1999921446=48, srv2042425784=54, srv7802002=89, srv1218453043=13, srv640162272=83, srv924635679=94, srv965351614=97, srv1145661967=11, srv1036582822=3, srv1529274003=27, srv1739917469=38, srv481893161=76, srv1030176332=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-24 12:53:15,797 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-24 12:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-24 12:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-24 12:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-24 12:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-24 12:53:15,798 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-24 12:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-24 12:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-24 12:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-24 12:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-24 12:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-24 12:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-24 12:53:15,799 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-24 12:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-24 12:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-24 12:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-24 12:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-24 12:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-24 12:53:15,800 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-24 12:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-24 12:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-24 12:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-24 12:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-24 12:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-24 12:53:15,801 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-24 12:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-24 12:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-24 12:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-24 12:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-24 12:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-24 12:53:15,802 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-24 12:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-24 12:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-24 12:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-24 12:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-24 12:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-24 12:53:15,803 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-24 12:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-24 12:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-24 12:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-24 12:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-24 12:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-24 12:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-24 12:53:15,804 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-24 12:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-24 12:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-24 12:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-24 12:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-24 12:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-24 12:53:15,805 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-24 12:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-24 12:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-24 12:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-24 12:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-24 12:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-24 12:53:15,806 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-24 12:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-24 12:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-24 12:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-24 12:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-24 12:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-24 12:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-24 12:53:15,807 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-24 12:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-24 12:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-24 12:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-24 12:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-24 12:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-24 12:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-24 12:53:15,808 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-24 12:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-24 12:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-24 12:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-24 12:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-24 12:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-24 12:53:15,809 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-24 12:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-24 12:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-24 12:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-24 12:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-24 12:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-24 12:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-24 12:53:15,810 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-24 12:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-24 12:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-24 12:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-24 12:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-24 12:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-24 12:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-24 12:53:15,811 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-24 12:53:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-24 12:53:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-24 12:53:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-24 12:53:15,812 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-24 12:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-24 12:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-24 12:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-24 12:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-24 12:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-24 12:53:15,813 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-24 12:53:15,814 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-24 12:53:16,131 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 12:53:16,132 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-24 12:53:16,132 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-24 12:53:16,575 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 729 ms to try 5000 different iterations.  Found a solution that moves 4694 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1815436328418507. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04127232019787021, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05584771662566684, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5215555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04127232019787021, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17716666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2395, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 12:53:16,586 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1138826844:80 , srv2022731423:81 , srv2114130530:82 , srv660641825:82 , srv1452993596:83 , srv1453974148:83 , srv1689410608:83 , srv332183356:83 , srv416385781:83 , srv640162272:83 , srv787981421:83 , srv1030176332:84 , srv1321539842:84 , srv1396845667:84 , srv1769524880:84 , srv488173331:84 , srv655602838:84 , srv726127470:84 , srv7802002:84 , srv947235495:84 , srv965351614:84 , srv101571443:85 , srv110243249:85 , srv1190469892:85 , srv1218453043:85 , srv165296794:85 , srv190868906:85 , srv1999921446:85 , srv2023859134:85 , srv2117043703:85 , srv2140567783:85 , srv218718837:85 , srv281638689:85 , srv341954614:85 , srv363814586:85 , srv481893161:85 , srv618859188:85 , srv729744960:85 , srv789886155:85 , srv935272039:85 , srv1138099888:86 , srv1369271437:86 , srv1669753075:86 , srv1672534903:86 , srv2042425784:86 , srv226031790:86 , srv307973174:86 , srv344945185:86 , srv521076890:86 , srv64428907:86 , srv96585853:86 , srv1090722484:87 , srv1090853466:87 , srv1377920783:87 , srv1380562253:87 , srv1386594810:87 , srv1496155809:87 , srv1522678840:87 , srv1680607619:87 , srv1752484082:87 , srv1922552583:87 , srv1967725111:87 , srv1989789908:87 , srv2005419637:87 , srv2036130442:87 , srv2036971151:87 , srv2099603350:87 , srv2112867828:87 , srv280005225:87 , srv308359903:87 , srv405424632:87 , srv483361315:87 , srv924635679:87 , srv1036582822:88 , srv106242799:88 , srv1374195671:88 , srv1511666074:88 , srv1531444564:88 , srv1992834715:88 , srv407757544:88 , srv1530162986:89 , srv1579442441:89 , srv1609017344:89 , srv1686040548:89 , srv1799378523:89 , srv1962139844:89 , srv439298455:89 , srv633271515:89 , srv798514929:89 , srv975769766:89 , srv1097346302:90 , srv139304918:90 , srv1529274003:90 , srv881671704:90 , srv1029394172:91 , srv1145661967:91 , srv1739917469:91 , srv333771445:91 , srv382249349:92 , srv502353201:457 }
2024-02-24 12:53:16,590 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-24 12:53:16,591 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1090722484=5, srv2005419637=49, srv947235495=96, srv363814586=70, srv344945185=69, srv1138099888=9, srv2023859134=51, srv1377920783=17, srv1989789908=46, srv1029394172=1, srv483361315=77, srv1967725111=45, srv1992834715=47, srv1369271437=15, srv341954614=68, srv281638689=63, srv2036971151=53, srv139304918=20, srv1669753075=33, srv1452993596=22, srv2112867828=56, srv190868906=42, srv502353201=79, srv2036130442=52, srv521076890=80, srv1769524880=40, srv881671704=93, srv1672534903=34, srv726127470=87, srv1380562253=18, srv382249349=71, srv407757544=73, srv280005225=62, srv332183356=66, srv935272039=95, srv1453974148=23, srv618859188=81, srv1680607619=35, srv308359903=65, srv1496155809=24, srv1799378523=41, srv1530162986=28, srv439298455=75, srv660641825=86, srv2022731423=50, srv110243249=8, srv787981421=90, srv1531444564=29, srv1097346302=7, srv789886155=91, srv975769766=99, srv1522678840=26, srv1190469892=12, srv1386594810=19, srv307973174=64, srv1609017344=31, srv1579442441=30, srv333771445=67, srv1138826844=10, srv633271515=82, srv1752484082=39, srv405424632=72, srv2099603350=55, srv1396845667=21, srv106242799=4, srv1686040548=36, srv226031790=61, srv1321539842=14, srv1511666074=25, srv165296794=32, srv64428907=84, srv1090853466=6, srv2114130530=57, srv101571443=0, srv1922552583=43, srv96585853=98, srv1962139844=44, srv218718837=60, srv655602838=85, srv488173331=78, srv1689410608=37, srv729744960=88, srv2117043703=58, srv416385781=74, srv1374195671=16, srv798514929=92, srv2140567783=59, srv1999921446=48, srv2042425784=54, srv7802002=89, srv1218453043=13, srv640162272=83, srv924635679=94, srv965351614=97, srv1145661967=11, srv1036582822=3, srv1529274003=27, srv1739917469=38, srv481893161=76, srv1030176332=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-24 12:53:16,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-24 12:53:16,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-24 12:53:16,614 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-24 12:53:16,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-24 12:53:16,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-24 12:53:16,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-24 12:53:16,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-24 12:53:16,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-24 12:53:16,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-24 12:53:16,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-24 12:53:16,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-24 12:53:16,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-24 12:53:16,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-24 12:53:16,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-24 12:53:16,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-24 12:53:16,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-24 12:53:16,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-24 12:53:16,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-24 12:53:16,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-24 12:53:16,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-24 12:53:16,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-24 12:53:16,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-24 12:53:16,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-24 12:53:16,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-24 12:53:16,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-24 12:53:16,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-24 12:53:16,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-24 12:53:16,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-24 12:53:16,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-24 12:53:16,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-24 12:53:16,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-24 12:53:16,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-24 12:53:16,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-24 12:53:16,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-24 12:53:16,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-24 12:53:16,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-24 12:53:16,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-24 12:53:16,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-24 12:53:16,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-24 12:53:16,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-24 12:53:16,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-24 12:53:16,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-24 12:53:16,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-24 12:53:16,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-24 12:53:16,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-24 12:53:16,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-24 12:53:16,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-24 12:53:16,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-24 12:53:16,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-24 12:53:16,622 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-24 12:53:16,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-24 12:53:16,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-24 12:53:16,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-24 12:53:16,623 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-24 12:53:16,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-24 12:53:16,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-24 12:53:16,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-24 12:53:16,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-24 12:53:16,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-24 12:53:16,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-24 12:53:16,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-24 12:53:16,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-24 12:53:16,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-24 12:53:16,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-24 12:53:16,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-24 12:53:16,624 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-24 12:53:16,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-24 12:53:16,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-24 12:53:16,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-24 12:53:16,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-24 12:53:16,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-24 12:53:16,625 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-24 12:53:16,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-24 12:53:16,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-24 12:53:16,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-24 12:53:16,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-24 12:53:16,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-24 12:53:16,626 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-24 12:53:16,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-24 12:53:16,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-24 12:53:16,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-24 12:53:16,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-24 12:53:16,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-24 12:53:16,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-24 12:53:16,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-24 12:53:16,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-24 12:53:16,627 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-24 12:53:16,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-24 12:53:16,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-24 12:53:16,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-24 12:53:16,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-24 12:53:16,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-24 12:53:16,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-24 12:53:16,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-24 12:53:16,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-24 12:53:16,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-24 12:53:16,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-24 12:53:16,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-24 12:53:16,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-24 12:53:16,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-24 12:53:16,628 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-24 12:53:16,783 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04127232019787021, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05584771662566684, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04127232019787021, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17716666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2395, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 12:53:16,783 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-24 12:53:16,784 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18151076475134692, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04127232019787021, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05584771662566684, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04127232019787021, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17716666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2395, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-24 12:53:17,151 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 496 ms to try 5000 different iterations.  Found a solution that moves 2625 regions; Going from a computed imbalance of 0.18151076475134692 to a new imbalance of 0.014306089079651937. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023853692130525357); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03354812948800065, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2916666666666667, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023853692130525357); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042416666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 12:53:17,164 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv881671704:79 , srv1029394172:80 , srv1097346302:80 , srv139304918:80 , srv1529274003:80 , srv382249349:80 , srv1145661967:81 , srv1374195671:81 , srv1579442441:81 , srv1739917469:81 , srv1799378523:81 , srv106242799:82 , srv2036130442:82 , srv633271515:82 , srv1511666074:83 , srv1530162986:83 , srv1962139844:83 , srv2036971151:83 , srv405424632:83 , srv407757544:83 , srv439298455:83 , srv975769766:83 , srv1036582822:84 , srv1369271437:84 , srv1377920783:84 , srv1380562253:84 , srv1496155809:84 , srv1531444564:84 , srv1609017344:84 , srv1686040548:84 , srv1992834715:84 , srv2042425784:84 , srv2112867828:84 , srv333771445:84 , srv798514929:84 , srv1090722484:85 , srv1090853466:85 , srv1190469892:85 , srv1218453043:85 , srv1522678840:85 , srv1752484082:85 , srv1989789908:85 , srv2005419637:85 , srv280005225:85 , srv308359903:85 , srv344945185:85 , srv363814586:85 , srv96585853:85 , srv1138099888:86 , srv1386594810:86 , srv165296794:86 , srv1680607619:86 , srv190868906:86 , srv1922552583:86 , srv2099603350:86 , srv218718837:86 , srv226031790:86 , srv307973174:86 , srv481893161:86 , srv64428907:86 , srv7802002:86 , srv924635679:86 , srv101571443:87 , srv110243249:87 , srv1321539842:87 , srv1669753075:87 , srv1672534903:87 , srv1769524880:87 , srv1967725111:87 , srv2022731423:87 , srv2140567783:87 , srv483361315:87 , srv521076890:87 , srv729744960:87 , srv789886155:87 , srv935272039:87 , srv1030176332:88 , srv1452993596:88 , srv1689410608:88 , srv2114130530:88 , srv416385781:88 , srv488173331:88 , srv618859188:88 , srv726127470:88 , srv1453974148:89 , srv1999921446:89 , srv2023859134:89 , srv2117043703:89 , srv341954614:89 , srv655602838:89 , srv640162272:90 , srv660641825:90 , srv947235495:90 , srv332183356:91 , srv787981421:91 , srv965351614:91 , srv1396845667:92 , srv281638689:92 , srv1138826844:93 , srv502353201:529 }
2024-02-24 12:53:17,170 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-24 12:53:17,171 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1090722484=5, srv2005419637=49, srv947235495=96, srv363814586=70, srv344945185=69, srv1138099888=9, srv2023859134=51, srv1377920783=17, srv1989789908=46, srv1029394172=1, srv483361315=77, srv1967725111=45, srv1992834715=47, srv1369271437=15, srv341954614=68, srv281638689=63, srv2036971151=53, srv139304918=20, srv1669753075=33, srv1452993596=22, srv2112867828=56, srv190868906=42, srv502353201=79, srv2036130442=52, srv521076890=80, srv1769524880=40, srv881671704=93, srv1672534903=34, srv726127470=87, srv1380562253=18, srv382249349=71, srv407757544=73, srv280005225=62, srv332183356=66, srv935272039=95, srv1453974148=23, srv618859188=81, srv1680607619=35, srv308359903=65, srv1496155809=24, srv1799378523=41, srv1530162986=28, srv439298455=75, srv660641825=86, srv2022731423=50, srv110243249=8, srv787981421=90, srv1531444564=29, srv1097346302=7, srv789886155=91, srv975769766=99, srv1522678840=26, srv1190469892=12, srv1386594810=19, srv307973174=64, srv1609017344=31, srv1579442441=30, srv333771445=67, srv1138826844=10, srv633271515=82, srv1752484082=39, srv405424632=72, srv2099603350=55, srv1396845667=21, srv106242799=4, srv1686040548=36, srv226031790=61, srv1321539842=14, srv1511666074=25, srv165296794=32, srv64428907=84, srv1090853466=6, srv2114130530=57, srv101571443=0, srv1922552583=43, srv96585853=98, srv1962139844=44, srv218718837=60, srv655602838=85, srv488173331=78, srv1689410608=37, srv729744960=88, srv2117043703=58, srv416385781=74, srv1374195671=16, srv798514929=92, srv2140567783=59, srv1999921446=48, srv2042425784=54, srv7802002=89, srv1218453043=13, srv640162272=83, srv924635679=94, srv965351614=97, srv1145661967=11, srv1036582822=3, srv1529274003=27, srv1739917469=38, srv481893161=76, srv1030176332=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-24 12:53:17,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-24 12:53:17,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-24 12:53:17,194 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-24 12:53:17,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-24 12:53:17,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-24 12:53:17,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-24 12:53:17,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-24 12:53:17,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-24 12:53:17,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-24 12:53:17,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-24 12:53:17,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-24 12:53:17,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-24 12:53:17,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-24 12:53:17,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-24 12:53:17,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-24 12:53:17,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-24 12:53:17,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-24 12:53:17,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-24 12:53:17,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-24 12:53:17,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-24 12:53:17,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-24 12:53:17,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-24 12:53:17,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-24 12:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-24 12:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-24 12:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-24 12:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-24 12:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-24 12:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-24 12:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-24 12:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-24 12:53:17,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-24 12:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-24 12:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-24 12:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-24 12:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-24 12:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-24 12:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-24 12:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-24 12:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-24 12:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-24 12:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-24 12:53:17,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-24 12:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-24 12:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-24 12:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-24 12:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-24 12:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-24 12:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-24 12:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-24 12:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-24 12:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-24 12:53:17,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-24 12:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-24 12:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-24 12:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-24 12:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-24 12:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-24 12:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-24 12:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-24 12:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-24 12:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-24 12:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-24 12:53:17,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-24 12:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-24 12:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-24 12:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-24 12:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-24 12:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-24 12:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-24 12:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-24 12:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-24 12:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-24 12:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-24 12:53:17,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-24 12:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-24 12:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-24 12:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-24 12:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-24 12:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-24 12:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-24 12:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-24 12:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-24 12:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-24 12:53:17,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-24 12:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-24 12:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-24 12:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-24 12:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-24 12:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-24 12:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-24 12:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-24 12:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-24 12:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-24 12:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-24 12:53:17,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-24 12:53:17,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-24 12:53:17,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-24 12:53:17,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-24 12:53:17,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-24 12:53:17,204 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-24 12:53:17,371 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023853692130525357); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03354812948800065, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023853692130525357); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042416666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 12:53:17,372 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-24 12:53:17,372 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014287708436794578, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023853692130525357); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03354812948800065, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023853692130525357); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042416666666666665, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-24 12:53:17,682 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 457 ms to try 5000 different iterations.  Found a solution that moves 972 regions; Going from a computed imbalance of 0.014287708436794578 to a new imbalance of 8.447340109921975E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006878417877522249); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007122465682345129); MoveCostFunction : (multiplier=7.0, imbalance=0.108, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006878417877522249); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 12:53:17,685 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv281638689:80 , srv1396845667:81 , srv965351614:82 , srv1967725111:83 , srv2023859134:83 , srv2117043703:83 , srv332183356:83 , srv341954614:83 , srv947235495:83 , srv1386594810:84 , srv1672534903:84 , srv2099603350:84 , srv333771445:84 , srv483361315:84 , srv521076890:84 , srv618859188:84 , srv655602838:84 , srv787981421:84 , srv798514929:84 , srv924635679:84 , srv101571443:85 , srv1030176332:85 , srv1036582822:85 , srv1090722484:85 , srv1090853466:85 , srv110243249:85 , srv1138099888:85 , srv1138826844:85 , srv1145661967:85 , srv1380562253:85 , srv1522678840:85 , srv1530162986:85 , srv1609017344:85 , srv1669753075:85 , srv1680607619:85 , srv1686040548:85 , srv1739917469:85 , srv1752484082:85 , srv1922552583:85 , srv1992834715:85 , srv1999921446:85 , srv2005419637:85 , srv226031790:85 , srv382249349:85 , srv488173331:85 , srv640162272:85 , srv726127470:85 , srv729744960:85 , srv789886155:85 , srv935272039:85 , srv975769766:85 , srv1321539842:86 , srv1452993596:86 , srv1453974148:86 , srv1496155809:86 , srv1511666074:86 , srv1531444564:86 , srv1689410608:86 , srv190868906:86 , srv1962139844:86 , srv1989789908:86 , srv2112867828:86 , srv2140567783:86 , srv218718837:86 , srv280005225:86 , srv307973174:86 , srv308359903:86 , srv416385781:86 , srv439298455:86 , srv633271515:86 , srv660641825:86 , srv96585853:86 , srv1029394172:87 , srv106242799:87 , srv1097346302:87 , srv1190469892:87 , srv1218453043:87 , srv1369271437:87 , srv1377920783:87 , srv1529274003:87 , srv1579442441:87 , srv1799378523:87 , srv2036971151:87 , srv2042425784:87 , srv2114130530:87 , srv344945185:87 , srv405424632:87 , srv407757544:87 , srv481893161:87 , srv64428907:87 , srv7802002:87 , srv139304918:88 , srv165296794:88 , srv1769524880:88 , srv363814586:88 , srv881671704:88 , srv1374195671:89 , srv2036130442:89 , srv2022731423:90 , srv502353201:533 }
2024-02-24 12:53:17,691 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-24 12:53:17,692 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1090722484=5, srv2005419637=49, srv947235495=96, srv363814586=70, srv344945185=69, srv1138099888=9, srv2023859134=51, srv1377920783=17, srv1989789908=46, srv1029394172=1, srv483361315=77, srv1967725111=45, srv1992834715=47, srv1369271437=15, srv341954614=68, srv281638689=63, srv2036971151=53, srv139304918=20, srv1669753075=33, srv1452993596=22, srv2112867828=56, srv190868906=42, srv502353201=79, srv2036130442=52, srv521076890=80, srv1769524880=40, srv881671704=93, srv1672534903=34, srv726127470=87, srv1380562253=18, srv382249349=71, srv407757544=73, srv280005225=62, srv332183356=66, srv935272039=95, srv1453974148=23, srv618859188=81, srv1680607619=35, srv308359903=65, srv1496155809=24, srv1799378523=41, srv1530162986=28, srv439298455=75, srv660641825=86, srv2022731423=50, srv110243249=8, srv787981421=90, srv1531444564=29, srv1097346302=7, srv789886155=91, srv975769766=99, srv1522678840=26, srv1190469892=12, srv1386594810=19, srv307973174=64, srv1609017344=31, srv1579442441=30, srv333771445=67, srv1138826844=10, srv633271515=82, srv1752484082=39, srv405424632=72, srv2099603350=55, srv1396845667=21, srv106242799=4, srv1686040548=36, srv226031790=61, srv1321539842=14, srv1511666074=25, srv165296794=32, srv64428907=84, srv1090853466=6, srv2114130530=57, srv101571443=0, srv1922552583=43, srv96585853=98, srv1962139844=44, srv218718837=60, srv655602838=85, srv488173331=78, srv1689410608=37, srv729744960=88, srv2117043703=58, srv416385781=74, srv1374195671=16, srv798514929=92, srv2140567783=59, srv1999921446=48, srv2042425784=54, srv7802002=89, srv1218453043=13, srv640162272=83, srv924635679=94, srv965351614=97, srv1145661967=11, srv1036582822=3, srv1529274003=27, srv1739917469=38, srv481893161=76, srv1030176332=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-24 12:53:17,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-24 12:53:17,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-24 12:53:17,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-24 12:53:17,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-24 12:53:17,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-24 12:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-24 12:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-24 12:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-24 12:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-24 12:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-24 12:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-24 12:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-24 12:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-24 12:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-24 12:53:17,712 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-24 12:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-24 12:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-24 12:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-24 12:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-24 12:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-24 12:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-24 12:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-24 12:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-24 12:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-24 12:53:17,713 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-24 12:53:17,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-24 12:53:17,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-24 12:53:17,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-24 12:53:17,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-24 12:53:17,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-24 12:53:17,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-24 12:53:17,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-24 12:53:17,714 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-24 12:53:17,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-24 12:53:17,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-24 12:53:17,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-24 12:53:17,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-24 12:53:17,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-24 12:53:17,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-24 12:53:17,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-24 12:53:17,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-24 12:53:17,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-24 12:53:17,715 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-24 12:53:17,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-24 12:53:17,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-24 12:53:17,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-24 12:53:17,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-24 12:53:17,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-24 12:53:17,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-24 12:53:17,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-24 12:53:17,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-24 12:53:17,716 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-24 12:53:17,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-24 12:53:17,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-24 12:53:17,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-24 12:53:17,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-24 12:53:17,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-24 12:53:17,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-24 12:53:17,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-24 12:53:17,717 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-24 12:53:17,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-24 12:53:17,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-24 12:53:17,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-24 12:53:17,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-24 12:53:17,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-24 12:53:17,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-24 12:53:17,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-24 12:53:17,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-24 12:53:17,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-24 12:53:17,718 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-24 12:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-24 12:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-24 12:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-24 12:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-24 12:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-24 12:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-24 12:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-24 12:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-24 12:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-24 12:53:17,719 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-24 12:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-24 12:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-24 12:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-24 12:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-24 12:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-24 12:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-24 12:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-24 12:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-24 12:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-24 12:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-24 12:53:17,720 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-24 12:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-24 12:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-24 12:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-24 12:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-24 12:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-24 12:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-24 12:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-24 12:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-24 12:53:17,721 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-24 12:53:17,722 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-24 12:53:17,880 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006878417877522249); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007122465682345129); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006878417877522249); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 12:53:17,881 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-02-24 12:53:17,881 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.379279215227303E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006878417877522249); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007122465682345129); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006878417877522249); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-02-24 12:53:18,239 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 501 ms to try 5000 different iterations.  Found a solution that moves 224 regions; Going from a computed imbalance of 8.379279215227303E-4 to a new imbalance of 1.1539851547288122E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.024888888888888887); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 12:53:18,240 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1769524880:83 , srv1999921446:83 , srv333771445:83 , srv64428907:83 , srv1029394172:84 , srv1138826844:84 , srv1369271437:84 , srv1374195671:84 , srv1377920783:84 , srv139304918:84 , srv1453974148:84 , srv1529274003:84 , srv1531444564:84 , srv1609017344:84 , srv165296794:84 , srv1669753075:84 , srv1680607619:84 , srv1686040548:84 , srv1922552583:84 , srv1962139844:84 , srv1989789908:84 , srv2022731423:84 , srv2023859134:84 , srv2036130442:84 , srv2036971151:84 , srv2140567783:84 , srv281638689:84 , srv307973174:84 , srv308359903:84 , srv344945185:84 , srv363814586:84 , srv407757544:84 , srv439298455:84 , srv481893161:84 , srv483361315:84 , srv521076890:84 , srv640162272:84 , srv660641825:84 , srv729744960:84 , srv787981421:84 , srv96585853:84 , srv101571443:85 , srv1030176332:85 , srv1036582822:85 , srv106242799:85 , srv1090722484:85 , srv1090853466:85 , srv1097346302:85 , srv110243249:85 , srv1138099888:85 , srv1145661967:85 , srv1190469892:85 , srv1218453043:85 , srv1321539842:85 , srv1380562253:85 , srv1386594810:85 , srv1396845667:85 , srv1452993596:85 , srv1496155809:85 , srv1511666074:85 , srv1522678840:85 , srv1530162986:85 , srv1579442441:85 , srv1739917469:85 , srv1752484082:85 , srv1799378523:85 , srv1967725111:85 , srv1992834715:85 , srv2005419637:85 , srv2042425784:85 , srv2099603350:85 , srv2112867828:85 , srv2114130530:85 , srv2117043703:85 , srv280005225:85 , srv332183356:85 , srv341954614:85 , srv382249349:85 , srv405424632:85 , srv416385781:85 , srv488173331:85 , srv633271515:85 , srv655602838:85 , srv726127470:85 , srv7802002:85 , srv789886155:85 , srv798514929:85 , srv881671704:85 , srv924635679:85 , srv935272039:85 , srv947235495:85 , srv965351614:85 , srv975769766:85 , srv1672534903:86 , srv1689410608:86 , srv190868906:86 , srv218718837:86 , srv226031790:86 , srv618859188:86 , srv502353201:624 }
2024-02-24 12:53:18,243 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-24 12:53:18,244 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1090722484=5, srv2005419637=49, srv947235495=96, srv363814586=70, srv344945185=69, srv1138099888=9, srv2023859134=51, srv1377920783=17, srv1989789908=46, srv1029394172=1, srv483361315=77, srv1967725111=45, srv1992834715=47, srv1369271437=15, srv341954614=68, srv281638689=63, srv2036971151=53, srv139304918=20, srv1669753075=33, srv1452993596=22, srv2112867828=56, srv190868906=42, srv502353201=79, srv2036130442=52, srv521076890=80, srv1769524880=40, srv881671704=93, srv1672534903=34, srv726127470=87, srv1380562253=18, srv382249349=71, srv407757544=73, srv280005225=62, srv332183356=66, srv935272039=95, srv1453974148=23, srv618859188=81, srv1680607619=35, srv308359903=65, srv1496155809=24, srv1799378523=41, srv1530162986=28, srv439298455=75, srv660641825=86, srv2022731423=50, srv110243249=8, srv787981421=90, srv1531444564=29, srv1097346302=7, srv789886155=91, srv975769766=99, srv1522678840=26, srv1190469892=12, srv1386594810=19, srv307973174=64, srv1609017344=31, srv1579442441=30, srv333771445=67, srv1138826844=10, srv633271515=82, srv1752484082=39, srv405424632=72, srv2099603350=55, srv1396845667=21, srv106242799=4, srv1686040548=36, srv226031790=61, srv1321539842=14, srv1511666074=25, srv165296794=32, srv64428907=84, srv1090853466=6, srv2114130530=57, srv101571443=0, srv1922552583=43, srv96585853=98, srv1962139844=44, srv218718837=60, srv655602838=85, srv488173331=78, srv1689410608=37, srv729744960=88, srv2117043703=58, srv416385781=74, srv1374195671=16, srv798514929=92, srv2140567783=59, srv1999921446=48, srv2042425784=54, srv7802002=89, srv1218453043=13, srv640162272=83, srv924635679=94, srv965351614=97, srv1145661967=11, srv1036582822=3, srv1529274003=27, srv1739917469=38, srv481893161=76, srv1030176332=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-24 12:53:18,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-24 12:53:18,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-24 12:53:18,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-24 12:53:18,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-24 12:53:18,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-24 12:53:18,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-24 12:53:18,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-24 12:53:18,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-24 12:53:18,255 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-24 12:53:18,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-02-24 12:53:18,257 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-02-24 12:53:18,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-02-24 12:53:18,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-02-24 12:53:18,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-02-24 12:53:18,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-02-24 12:53:18,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-02-24 12:53:18,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-02-24 12:53:18,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-02-24 12:53:18,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-02-24 12:53:18,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-02-24 12:53:18,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-02-24 12:53:18,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-02-24 12:53:18,258 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-02-24 12:53:18,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-02-24 12:53:18,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-02-24 12:53:18,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-02-24 12:53:18,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-02-24 12:53:18,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-02-24 12:53:18,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-02-24 12:53:18,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-02-24 12:53:18,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-02-24 12:53:18,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-02-24 12:53:18,259 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-02-24 12:53:18,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-02-24 12:53:18,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-02-24 12:53:18,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-02-24 12:53:18,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-02-24 12:53:18,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-02-24 12:53:18,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-02-24 12:53:18,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-02-24 12:53:18,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-02-24 12:53:18,260 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-02-24 12:53:18,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-02-24 12:53:18,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-02-24 12:53:18,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-02-24 12:53:18,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-02-24 12:53:18,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-02-24 12:53:18,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-02-24 12:53:18,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-02-24 12:53:18,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-02-24 12:53:18,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-02-24 12:53:18,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-02-24 12:53:18,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-02-24 12:53:18,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-02-24 12:53:18,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-02-24 12:53:18,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-02-24 12:53:18,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-02-24 12:53:18,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-02-24 12:53:18,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-02-24 12:53:18,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-02-24 12:53:18,262 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-02-24 12:53:18,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-02-24 12:53:18,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-02-24 12:53:18,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-02-24 12:53:18,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-02-24 12:53:18,367 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=9.971370023460306E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.737793696791342E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.737793696791342E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 12:53:18,368 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1769524880:83 , srv1999921446:83 , srv333771445:83 , srv64428907:83 , srv1029394172:84 , srv1138826844:84 , srv1369271437:84 , srv1374195671:84 , srv1377920783:84 , srv139304918:84 , srv1453974148:84 , srv1529274003:84 , srv1531444564:84 , srv1609017344:84 , srv165296794:84 , srv1669753075:84 , srv1680607619:84 , srv1686040548:84 , srv1922552583:84 , srv1962139844:84 , srv1989789908:84 , srv2022731423:84 , srv2023859134:84 , srv2036130442:84 , srv2036971151:84 , srv2140567783:84 , srv281638689:84 , srv307973174:84 , srv308359903:84 , srv344945185:84 , srv363814586:84 , srv407757544:84 , srv439298455:84 , srv481893161:84 , srv483361315:84 , srv521076890:84 , srv640162272:84 , srv660641825:84 , srv729744960:84 , srv787981421:84 , srv96585853:84 , srv101571443:85 , srv1030176332:85 , srv1036582822:85 , srv106242799:85 , srv1090722484:85 , srv1090853466:85 , srv1097346302:85 , srv110243249:85 , srv1138099888:85 , srv1145661967:85 , srv1190469892:85 , srv1218453043:85 , srv1321539842:85 , srv1380562253:85 , srv1386594810:85 , srv1396845667:85 , srv1452993596:85 , srv1496155809:85 , srv1511666074:85 , srv1522678840:85 , srv1530162986:85 , srv1579442441:85 , srv1739917469:85 , srv1752484082:85 , srv1799378523:85 , srv1967725111:85 , srv1992834715:85 , srv2005419637:85 , srv2042425784:85 , srv2099603350:85 , srv2112867828:85 , srv2114130530:85 , srv2117043703:85 , srv280005225:85 , srv332183356:85 , srv341954614:85 , srv382249349:85 , srv405424632:85 , srv416385781:85 , srv488173331:85 , srv633271515:85 , srv655602838:85 , srv726127470:85 , srv7802002:85 , srv789886155:85 , srv798514929:85 , srv881671704:85 , srv924635679:85 , srv935272039:85 , srv947235495:85 , srv965351614:85 , srv975769766:85 , srv1672534903:86 , srv1689410608:86 , srv190868906:86 , srv218718837:86 , srv226031790:86 , srv618859188:86 , srv502353201:624 }
2024-02-24 12:53:18,393 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=292 (was 248) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=21215 (was 21945)
2024-02-24 12:53:18,402 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=292, ProcessCount=171, AvailableMemoryMB=21215
2024-02-24 12:53:18,403 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-02-24 12:53:18,403 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-02-24 12:53:18,419 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1008127067:84 , srv1256147039:84 , srv1426759196:84 , srv1447957356:84 , srv1504954007:84 , srv1517110723:84 , srv1667337737:84 , srv1683099434:84 , srv1721612927:84 , srv1811695088:84 , srv1841838024:84 , srv1903422088:84 , srv2020782383:84 , srv2023482113:84 , srv2045788031:84 , srv208549645:84 , srv2113603515:84 , srv223425128:84 , srv246054690:84 , srv249365363:84 , srv279834048:84 , srv472578804:84 , srv500428799:84 , srv502279537:84 , srv626149199:84 , srv657407582:84 , srv69560476:84 , srv908868364:84 , srv982458146:84 , srv797670363:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-02-24 12:53:18,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-24 12:53:18,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv69560476=26, srv1721612927=8, srv1667337737=6, srv1903422088=11, srv1256147039=1, srv1008127067=0, srv2023482113=13, srv279834048=20, srv472578804=21, srv908868364=28, srv500428799=22, srv1504954007=4, srv1447957356=3, srv657407582=25, srv1811695088=9, srv2020782383=12, srv249365363=19, srv502279537=23, srv2045788031=14, srv246054690=18, srv982458146=29, srv626149199=24, srv797670363=27, srv1426759196=2, srv2113603515=16, srv223425128=17, srv1683099434=7, srv208549645=15, srv1517110723=5, srv1841838024=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-24 12:53:18,431 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-24 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-24 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-24 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-24 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-24 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-24 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-24 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-24 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-24 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-24 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-24 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-24 12:53:18,432 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-02-24 12:53:18,448 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1469992289695676, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 12:53:18,449 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912656233034796, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1469992289695676, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-02-24 12:55:18,435 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5557221 different iterations.  Found a solution that moves 1876 regions; Going from a computed imbalance of 0.9912656233034796 to a new imbalance of 4.37867758735265E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6948148148148148, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 12:55:18,440 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1008127067:90 , srv1256147039:90 , srv1426759196:90 , srv1447957356:90 , srv1504954007:90 , srv1517110723:90 , srv1667337737:90 , srv1683099434:90 , srv1721612927:90 , srv1811695088:90 , srv1841838024:90 , srv1903422088:90 , srv2020782383:90 , srv2023482113:90 , srv2045788031:90 , srv208549645:90 , srv2113603515:90 , srv223425128:90 , srv246054690:90 , srv249365363:90 , srv279834048:90 , srv472578804:90 , srv500428799:90 , srv502279537:90 , srv626149199:90 , srv657407582:90 , srv69560476:90 , srv797670363:90 , srv908868364:90 , srv982458146:90 }
2024-02-24 12:55:18,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-02-24 12:55:18,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv69560476=26, srv1721612927=8, srv1667337737=6, srv1903422088=11, srv1256147039=1, srv1008127067=0, srv2023482113=13, srv279834048=20, srv472578804=21, srv908868364=28, srv500428799=22, srv1504954007=4, srv1447957356=3, srv657407582=25, srv1811695088=9, srv2020782383=12, srv249365363=19, srv502279537=23, srv2045788031=14, srv246054690=18, srv982458146=29, srv626149199=24, srv797670363=27, srv1426759196=2, srv2113603515=16, srv223425128=17, srv1683099434=7, srv208549645=15, srv1517110723=5, srv1841838024=10} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-02-24 12:55:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-02-24 12:55:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-02-24 12:55:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-02-24 12:55:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-02-24 12:55:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-02-24 12:55:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-02-24 12:55:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-02-24 12:55:18,445 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-02-24 12:55:18,446 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-02-24 12:55:18,447 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-02-24 12:55:18,460 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-02-24 12:55:18,460 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1008127067:90 , srv1256147039:90 , srv1426759196:90 , srv1447957356:90 , srv1504954007:90 , srv1517110723:90 , srv1667337737:90 , srv1683099434:90 , srv1721612927:90 , srv1811695088:90 , srv1841838024:90 , srv1903422088:90 , srv2020782383:90 , srv2023482113:90 , srv2045788031:90 , srv208549645:90 , srv2113603515:90 , srv223425128:90 , srv246054690:90 , srv249365363:90 , srv279834048:90 , srv472578804:90 , srv500428799:90 , srv502279537:90 , srv626149199:90 , srv657407582:90 , srv69560476:90 , srv797670363:90 , srv908868364:90 , srv982458146:90 }
2024-02-24 12:55:18,474 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=209 (was 292), ProcessCount=168 (was 171), AvailableMemoryMB=20761 (was 21215)