2023-10-24 16:53:26,014 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-10-24 16:53:26,327 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-10-24 16:53:26,449 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-10-24 16:53:26,513 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-24 16:53:26,520 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-10-24 16:53:26,533 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=357, ProcessCount=170, AvailableMemoryMB=8613
2023-10-24 16:53:26,535 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-24 16:53:26,536 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-10-24 16:53:26,662 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1000791352:84 , srv1003346276:84 , srv1016501949:84 , srv1029052626:84 , srv1036020276:84 , srv1085717676:84 , srv1106284851:84 , srv1108890087:84 , srv1135265785:84 , srv1152816150:84 , srv1170626615:84 , srv1206004521:84 , srv1241062912:84 , srv1247905138:84 , srv1287600987:84 , srv129123913:84 , srv1316972006:84 , srv1350739399:84 , srv1371670363:84 , srv139110028:84 , srv1407800120:84 , srv1409989776:84 , srv1434181857:84 , srv1444612878:84 , srv1445478585:84 , srv1463507349:84 , srv1477690784:84 , srv1497672333:84 , srv1522976858:84 , srv1532585432:84 , srv1561405676:84 , srv1576034011:84 , srv1578499947:84 , srv1605058400:84 , srv1645724857:84 , srv1665593950:84 , srv1678834067:84 , srv1680449534:84 , srv1683286740:84 , srv1686543677:84 , srv1708873842:84 , srv1715591884:84 , srv1719022876:84 , srv1720167746:84 , srv1752215766:84 , srv1763529663:84 , srv1770625145:84 , srv1773600044:84 , srv1777274885:84 , srv178471067:84 , srv1789669963:84 , srv1790815033:84 , srv1851825723:84 , srv1866803148:84 , srv1882940550:84 , srv1899775889:84 , srv193196115:84 , srv1975484373:84 , srv198653677:84 , srv1991560165:84 , srv2014106704:84 , srv2022456489:84 , srv2041530587:84 , srv2056624196:84 , srv2059539672:84 , srv2073804675:84 , srv210458165:84 , srv2113368031:84 , srv2123482192:84 , srv233427243:84 , srv285612114:84 , srv296343021:84 , srv386886220:84 , srv396977076:84 , srv425591188:84 , srv428340976:84 , srv444438843:84 , srv449257017:84 , srv498271219:84 , srv582636097:84 , srv626721289:84 , srv636641587:84 , srv653696144:84 , srv661307462:84 , srv695019410:84 , srv713027587:84 , srv744188861:84 , srv7476667:84 , srv762387551:84 , srv781938911:84 , srv785830970:84 , srv802650834:84 , srv825836484:84 , srv827433754:84 , srv852961082:84 , srv87102211:84 , srv873571285:84 , srv911212690:84 , srv916248920:84 , srv1141876085:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-10-24 16:53:26,670 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-24 16:53:26,677 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1752215766=45, srv1029052626=3, srv1170626615=11, srv1882940550=55, srv1715591884=42, srv1678834067=37, srv827433754=94, srv1790815033=52, srv386886220=73, srv2041530587=63, srv1247905138=14, srv1463507349=26, srv1773600044=48, srv2073804675=66, srv7476667=88, srv396977076=74, srv1108890087=7, srv1561405676=31, srv653696144=83, srv1152816150=10, srv296343021=72, srv873571285=97, srv1371670363=19, srv1445478585=25, srv1576034011=32, srv695019410=85, srv1407800120=21, srv825836484=93, srv1477690784=27, srv1680449534=38, srv425591188=75, srv1241062912=13, srv636641587=82, srv762387551=89, srv911212690=98, srv1206004521=12, srv1645724857=35, srv916248920=99, srv1434181857=23, srv1686543677=40, srv1522976858=29, srv1719022876=43, srv178471067=50, srv1444612878=24, srv1578499947=33, srv1991560165=60, srv139110028=20, srv1975484373=58, srv744188861=87, srv802650834=92, srv129123913=16, srv2123482192=69, srv193196115=57, srv1899775889=56, srv626721289=81, srv285612114=71, srv444438843=77, srv1106284851=6, srv210458165=67, srv1605058400=34, srv661307462=84, srv852961082=95, srv1016501949=2, srv498271219=79, srv2059539672=65, srv1036020276=4, srv1497672333=28, srv1085717676=5, srv2022456489=62, srv1665593950=36, srv2113368031=68, srv87102211=96, srv2014106704=61, srv1866803148=54, srv1316972006=17, srv233427243=70, srv1141876085=9, srv1000791352=0, srv713027587=86, srv1003346276=1, srv2056624196=64, srv1532585432=30, srv1851825723=53, srv1720167746=44, srv449257017=78, srv1789669963=51, srv1683286740=39, srv428340976=76, srv1763529663=46, srv582636097=80, srv1770625145=47, srv781938911=90, srv1135265785=8, srv785830970=91, srv1350739399=18, srv1777274885=49, srv1708873842=41, srv1409989776=22, srv198653677=59, srv1287600987=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-24 16:53:26,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-24 16:53:26,775 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-24 16:53:26,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-24 16:53:26,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-24 16:53:26,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-24 16:53:26,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-24 16:53:26,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-24 16:53:26,776 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-24 16:53:26,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-24 16:53:26,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-24 16:53:26,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-24 16:53:26,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-24 16:53:26,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-24 16:53:26,777 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-24 16:53:26,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-24 16:53:26,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-24 16:53:26,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-24 16:53:26,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-24 16:53:26,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-24 16:53:26,778 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-24 16:53:26,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-24 16:53:26,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-24 16:53:26,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-24 16:53:26,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-24 16:53:26,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-24 16:53:26,779 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-24 16:53:26,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-24 16:53:26,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-24 16:53:26,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-24 16:53:26,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-24 16:53:26,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-24 16:53:26,780 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-24 16:53:26,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-24 16:53:26,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-24 16:53:26,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-24 16:53:26,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-24 16:53:26,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-24 16:53:26,781 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-24 16:53:26,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-24 16:53:26,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-24 16:53:26,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-24 16:53:26,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-24 16:53:26,782 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-24 16:53:26,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-24 16:53:26,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-24 16:53:26,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-24 16:53:26,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-24 16:53:26,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-24 16:53:26,783 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-24 16:53:26,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-24 16:53:26,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-24 16:53:26,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-24 16:53:26,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-24 16:53:26,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-24 16:53:26,784 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-24 16:53:26,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-24 16:53:26,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-24 16:53:26,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-24 16:53:26,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-24 16:53:26,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-24 16:53:26,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-24 16:53:26,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-24 16:53:26,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-24 16:53:26,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-24 16:53:26,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-24 16:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-24 16:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-24 16:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-24 16:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-24 16:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-24 16:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-24 16:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-24 16:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-24 16:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-24 16:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-24 16:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-24 16:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-24 16:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-24 16:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-24 16:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-24 16:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-24 16:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-24 16:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-24 16:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-24 16:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-24 16:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-24 16:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-24 16:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-24 16:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-24 16:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-24 16:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-24 16:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-24 16:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-24 16:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-24 16:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-24 16:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-24 16:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-24 16:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-24 16:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-24 16:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-24 16:53:26,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-24 16:53:27,074 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-24 16:53:27,074 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-24 16:53:27,074 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-24 16:53:27,468 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 658 ms to try 5000 different iterations.  Found a solution that moves 4689 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.17953804316545843. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039649606343599354, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053708695924937065, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.521, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039649606343599354, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17475, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24158333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-24 16:53:27,482 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1708873842:81 , srv661307462:81 , srv911212690:81 , srv1350739399:82 , srv1522976858:82 , srv1645724857:82 , srv1683286740:82 , srv1719022876:82 , srv1899775889:82 , srv1975484373:82 , srv1991560165:82 , srv233427243:82 , srv449257017:82 , srv636641587:82 , srv713027587:82 , srv7476667:82 , srv87102211:82 , srv1789669963:83 , srv1882940550:83 , srv2056624196:83 , srv762387551:83 , srv1029052626:84 , srv1247905138:84 , srv1371670363:84 , srv1477690784:84 , srv1773600044:84 , srv178471067:84 , srv193196115:84 , srv210458165:84 , srv425591188:84 , srv781938911:84 , srv827433754:84 , srv1000791352:85 , srv1016501949:85 , srv1108890087:85 , srv1287600987:85 , srv1316972006:85 , srv1409989776:85 , srv1444612878:85 , srv1561405676:85 , srv1665593950:85 , srv1678834067:85 , srv285612114:85 , srv296343021:85 , srv396977076:85 , srv744188861:85 , srv1036020276:86 , srv1106284851:86 , srv1407800120:86 , srv1777274885:86 , srv1851825723:86 , srv198653677:86 , srv2113368031:86 , srv2123482192:86 , srv785830970:86 , srv852961082:86 , srv1135265785:87 , srv1152816150:87 , srv1206004521:87 , srv1434181857:87 , srv1605058400:87 , srv1680449534:87 , srv1715591884:87 , srv1763529663:87 , srv1770625145:87 , srv2022456489:87 , srv2059539672:87 , srv386886220:87 , srv498271219:87 , srv582636097:87 , srv626721289:87 , srv916248920:87 , srv1085717676:88 , srv1445478585:88 , srv1463507349:88 , srv1866803148:88 , srv802650834:88 , srv1170626615:89 , srv1241062912:89 , srv1497672333:89 , srv1532585432:89 , srv1686543677:89 , srv653696144:89 , srv139110028:90 , srv1578499947:90 , srv1720167746:90 , srv1790815033:90 , srv2041530587:90 , srv825836484:90 , srv129123913:92 , srv2073804675:92 , srv428340976:92 , srv695019410:92 , srv1003346276:94 , srv444438843:95 , srv1576034011:98 , srv1752215766:98 , srv873571285:100 , srv2014106704:103 , srv1141876085:441 }
2023-10-24 16:53:27,486 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-24 16:53:27,487 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1752215766=45, srv1029052626=3, srv1170626615=11, srv1882940550=55, srv1715591884=42, srv1678834067=37, srv827433754=94, srv1790815033=52, srv386886220=73, srv2041530587=63, srv1247905138=14, srv1463507349=26, srv1773600044=48, srv2073804675=66, srv7476667=88, srv396977076=74, srv1108890087=7, srv1561405676=31, srv653696144=83, srv1152816150=10, srv296343021=72, srv873571285=97, srv1371670363=19, srv1445478585=25, srv1576034011=32, srv695019410=85, srv1407800120=21, srv825836484=93, srv1477690784=27, srv1680449534=38, srv425591188=75, srv1241062912=13, srv636641587=82, srv762387551=89, srv911212690=98, srv1206004521=12, srv1645724857=35, srv916248920=99, srv1434181857=23, srv1686543677=40, srv1522976858=29, srv1719022876=43, srv178471067=50, srv1444612878=24, srv1578499947=33, srv1991560165=60, srv139110028=20, srv1975484373=58, srv744188861=87, srv802650834=92, srv129123913=16, srv2123482192=69, srv193196115=57, srv1899775889=56, srv626721289=81, srv285612114=71, srv444438843=77, srv1106284851=6, srv210458165=67, srv1605058400=34, srv661307462=84, srv852961082=95, srv1016501949=2, srv498271219=79, srv2059539672=65, srv1036020276=4, srv1497672333=28, srv1085717676=5, srv2022456489=62, srv1665593950=36, srv2113368031=68, srv87102211=96, srv2014106704=61, srv1866803148=54, srv1316972006=17, srv233427243=70, srv1141876085=9, srv1000791352=0, srv713027587=86, srv1003346276=1, srv2056624196=64, srv1532585432=30, srv1851825723=53, srv1720167746=44, srv449257017=78, srv1789669963=51, srv1683286740=39, srv428340976=76, srv1763529663=46, srv582636097=80, srv1770625145=47, srv781938911=90, srv1135265785=8, srv785830970=91, srv1350739399=18, srv1777274885=49, srv1708873842=41, srv1409989776=22, srv198653677=59, srv1287600987=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-24 16:53:27,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-24 16:53:27,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-24 16:53:27,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-24 16:53:27,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-24 16:53:27,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-24 16:53:27,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-24 16:53:27,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-24 16:53:27,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-24 16:53:27,504 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-24 16:53:27,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-24 16:53:27,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-24 16:53:27,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-24 16:53:27,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-24 16:53:27,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-24 16:53:27,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-24 16:53:27,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-24 16:53:27,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-24 16:53:27,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-24 16:53:27,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-24 16:53:27,505 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-24 16:53:27,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-24 16:53:27,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-24 16:53:27,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-24 16:53:27,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-24 16:53:27,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-24 16:53:27,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-24 16:53:27,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-24 16:53:27,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-24 16:53:27,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-24 16:53:27,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-24 16:53:27,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-24 16:53:27,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-24 16:53:27,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-24 16:53:27,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-24 16:53:27,506 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-24 16:53:27,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-24 16:53:27,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-24 16:53:27,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-24 16:53:27,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-24 16:53:27,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-24 16:53:27,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-24 16:53:27,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-24 16:53:27,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-24 16:53:27,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-24 16:53:27,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-24 16:53:27,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-24 16:53:27,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-24 16:53:27,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-24 16:53:27,507 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-24 16:53:27,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-24 16:53:27,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-24 16:53:27,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-24 16:53:27,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-24 16:53:27,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-24 16:53:27,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-24 16:53:27,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-24 16:53:27,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-24 16:53:27,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-24 16:53:27,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-24 16:53:27,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-24 16:53:27,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-24 16:53:27,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-24 16:53:27,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-24 16:53:27,508 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-24 16:53:27,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-24 16:53:27,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-24 16:53:27,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-24 16:53:27,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-24 16:53:27,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-24 16:53:27,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-24 16:53:27,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-24 16:53:27,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-24 16:53:27,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-24 16:53:27,509 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-24 16:53:27,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-24 16:53:27,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-24 16:53:27,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-24 16:53:27,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-24 16:53:27,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-24 16:53:27,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-24 16:53:27,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-24 16:53:27,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-24 16:53:27,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-24 16:53:27,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-24 16:53:27,510 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-24 16:53:27,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-24 16:53:27,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-24 16:53:27,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-24 16:53:27,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-24 16:53:27,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-24 16:53:27,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-24 16:53:27,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-24 16:53:27,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-24 16:53:27,511 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-24 16:53:27,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-24 16:53:27,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-24 16:53:27,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-24 16:53:27,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-24 16:53:27,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-24 16:53:27,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-24 16:53:27,512 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-24 16:53:27,685 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039649606343599354, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053708695924937065, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039649606343599354, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17475, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24158333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-24 16:53:27,685 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-24 16:53:27,685 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17950521008570294, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039649606343599354, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.053708695924937065, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039649606343599354, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17475, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24158333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-24 16:53:28,000 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 460 ms to try 5000 different iterations.  Found a solution that moves 2698 regions; Going from a computed imbalance of 0.17950521008570294 to a new imbalance of 0.015001247472616268. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.018389449199895727); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.028713531498817026, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29977777777777775, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.018389449199895727); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0465, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-24 16:53:28,012 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2014106704:67 , srv1576034011:74 , srv1752215766:77 , srv444438843:77 , srv1003346276:78 , srv129123913:78 , srv873571285:78 , srv428340976:79 , srv2073804675:80 , srv1445478585:81 , srv1497672333:81 , srv1720167746:81 , srv1170626615:82 , srv139110028:82 , srv1578499947:82 , srv1790815033:82 , srv1605058400:83 , srv2041530587:83 , srv802650834:83 , srv1152816150:84 , srv1444612878:84 , srv1532585432:84 , srv1686543677:84 , srv1770625145:84 , srv1777274885:84 , srv2059539672:84 , srv386886220:84 , srv498271219:84 , srv653696144:84 , srv695019410:84 , srv825836484:84 , srv1085717676:85 , srv1135265785:85 , srv1241062912:85 , srv1407800120:85 , srv1680449534:85 , srv1763529663:85 , srv1851825723:85 , srv198653677:85 , srv2022456489:85 , srv2113368031:85 , srv582636097:85 , srv852961082:85 , srv1106284851:86 , srv1206004521:86 , srv2123482192:86 , srv626721289:86 , srv785830970:86 , srv1678834067:87 , srv1683286740:87 , srv1715591884:87 , srv178471067:87 , srv285612114:87 , srv744188861:87 , srv762387551:87 , srv916248920:87 , srv1029052626:88 , srv1036020276:88 , srv1287600987:88 , srv1316972006:88 , srv1409989776:88 , srv1434181857:88 , srv1463507349:88 , srv1477690784:88 , srv1522976858:88 , srv1665593950:88 , srv1882940550:88 , srv781938911:88 , srv87102211:88 , srv1016501949:89 , srv1108890087:89 , srv1247905138:89 , srv1371670363:89 , srv1561405676:89 , srv1708873842:89 , srv1773600044:89 , srv1789669963:89 , srv1866803148:89 , srv193196115:89 , srv2056624196:89 , srv296343021:89 , srv396977076:89 , srv425591188:89 , srv449257017:89 , srv661307462:89 , srv827433754:89 , srv1000791352:90 , srv1899775889:90 , srv1975484373:90 , srv1350739399:91 , srv1645724857:91 , srv1719022876:91 , srv1991560165:91 , srv210458165:91 , srv636641587:91 , srv7476667:91 , srv713027587:92 , srv233427243:93 , srv911212690:93 , srv1141876085:496 }
2023-10-24 16:53:28,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-24 16:53:28,017 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1752215766=45, srv1029052626=3, srv1170626615=11, srv1882940550=55, srv1715591884=42, srv1678834067=37, srv827433754=94, srv1790815033=52, srv386886220=73, srv2041530587=63, srv1247905138=14, srv1463507349=26, srv1773600044=48, srv2073804675=66, srv7476667=88, srv396977076=74, srv1108890087=7, srv1561405676=31, srv653696144=83, srv1152816150=10, srv296343021=72, srv873571285=97, srv1371670363=19, srv1445478585=25, srv1576034011=32, srv695019410=85, srv1407800120=21, srv825836484=93, srv1477690784=27, srv1680449534=38, srv425591188=75, srv1241062912=13, srv636641587=82, srv762387551=89, srv911212690=98, srv1206004521=12, srv1645724857=35, srv916248920=99, srv1434181857=23, srv1686543677=40, srv1522976858=29, srv1719022876=43, srv178471067=50, srv1444612878=24, srv1578499947=33, srv1991560165=60, srv139110028=20, srv1975484373=58, srv744188861=87, srv802650834=92, srv129123913=16, srv2123482192=69, srv193196115=57, srv1899775889=56, srv626721289=81, srv285612114=71, srv444438843=77, srv1106284851=6, srv210458165=67, srv1605058400=34, srv661307462=84, srv852961082=95, srv1016501949=2, srv498271219=79, srv2059539672=65, srv1036020276=4, srv1497672333=28, srv1085717676=5, srv2022456489=62, srv1665593950=36, srv2113368031=68, srv87102211=96, srv2014106704=61, srv1866803148=54, srv1316972006=17, srv233427243=70, srv1141876085=9, srv1000791352=0, srv713027587=86, srv1003346276=1, srv2056624196=64, srv1532585432=30, srv1851825723=53, srv1720167746=44, srv449257017=78, srv1789669963=51, srv1683286740=39, srv428340976=76, srv1763529663=46, srv582636097=80, srv1770625145=47, srv781938911=90, srv1135265785=8, srv785830970=91, srv1350739399=18, srv1777274885=49, srv1708873842=41, srv1409989776=22, srv198653677=59, srv1287600987=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-24 16:53:28,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-24 16:53:28,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-24 16:53:28,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-24 16:53:28,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-24 16:53:28,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-24 16:53:28,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-24 16:53:28,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-24 16:53:28,041 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-24 16:53:28,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-24 16:53:28,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-24 16:53:28,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-24 16:53:28,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-24 16:53:28,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-24 16:53:28,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-24 16:53:28,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-24 16:53:28,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-24 16:53:28,042 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-24 16:53:28,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-24 16:53:28,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-24 16:53:28,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-24 16:53:28,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-24 16:53:28,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-24 16:53:28,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-24 16:53:28,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-24 16:53:28,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-24 16:53:28,043 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-24 16:53:28,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-24 16:53:28,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-24 16:53:28,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-24 16:53:28,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-24 16:53:28,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-24 16:53:28,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-24 16:53:28,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-24 16:53:28,044 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-24 16:53:28,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-24 16:53:28,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-24 16:53:28,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-24 16:53:28,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-24 16:53:28,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-24 16:53:28,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-24 16:53:28,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-24 16:53:28,045 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-24 16:53:28,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-24 16:53:28,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-24 16:53:28,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-24 16:53:28,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-24 16:53:28,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-24 16:53:28,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-24 16:53:28,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-24 16:53:28,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-24 16:53:28,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-24 16:53:28,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-24 16:53:28,046 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-24 16:53:28,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-24 16:53:28,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-24 16:53:28,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-24 16:53:28,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-24 16:53:28,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-24 16:53:28,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-24 16:53:28,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-24 16:53:28,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-24 16:53:28,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-24 16:53:28,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-24 16:53:28,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-24 16:53:28,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-24 16:53:28,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-24 16:53:28,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-24 16:53:28,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-24 16:53:28,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-24 16:53:28,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-24 16:53:28,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-24 16:53:28,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-24 16:53:28,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-24 16:53:28,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-24 16:53:28,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-24 16:53:28,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-24 16:53:28,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-24 16:53:28,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-24 16:53:28,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-24 16:53:28,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-24 16:53:28,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-24 16:53:28,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-24 16:53:28,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-24 16:53:28,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-24 16:53:28,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-24 16:53:28,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-24 16:53:28,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-24 16:53:28,049 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-24 16:53:28,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-24 16:53:28,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-24 16:53:28,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-24 16:53:28,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-24 16:53:28,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-24 16:53:28,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-24 16:53:28,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-24 16:53:28,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-24 16:53:28,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-24 16:53:28,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-24 16:53:28,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-24 16:53:28,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-24 16:53:28,050 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-24 16:53:28,211 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.018389449199895727); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.028713531498817026, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.018389449199895727); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0465, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-24 16:53:28,212 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-24 16:53:28,212 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014982355672833734, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.018389449199895727); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.028713531498817026, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.018389449199895727); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01175); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0465, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-24 16:53:28,589 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 516 ms to try 5000 different iterations.  Found a solution that moves 1026 regions; Going from a computed imbalance of 0.014982355672833734 to a new imbalance of 6.814423372553137E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0044444444444444444); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00670025210172808); MoveCostFunction : (multiplier=7.0, imbalance=0.114, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0044444444444444444); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-24 16:53:28,591 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv873571285:79 , srv1463507349:81 , srv1866803148:81 , srv1434181857:82 , srv210458165:82 , srv233427243:82 , srv1036020276:83 , srv1108890087:83 , srv1561405676:83 , srv1752215766:83 , srv695019410:83 , srv825836484:83 , srv911212690:83 , srv1000791352:84 , srv1206004521:84 , srv1241062912:84 , srv1287600987:84 , srv1715591884:84 , srv1773600044:84 , srv1991560165:84 , srv285612114:84 , srv296343021:84 , srv396977076:84 , srv425591188:84 , srv582636097:84 , srv636641587:84 , srv653696144:84 , srv916248920:84 , srv1003346276:85 , srv1016501949:85 , srv1029052626:85 , srv1085717676:85 , srv1247905138:85 , srv1316972006:85 , srv1350739399:85 , srv1371670363:85 , srv139110028:85 , srv1409989776:85 , srv1477690784:85 , srv1532585432:85 , srv1576034011:85 , srv1578499947:85 , srv1645724857:85 , srv1665593950:85 , srv1678834067:85 , srv1686543677:85 , srv1719022876:85 , srv1789669963:85 , srv1790815033:85 , srv1899775889:85 , srv193196115:85 , srv198653677:85 , srv2022456489:85 , srv2041530587:85 , srv444438843:85 , srv626721289:85 , srv713027587:85 , srv7476667:85 , srv781938911:85 , srv827433754:85 , srv1135265785:86 , srv1170626615:86 , srv1407800120:86 , srv1680449534:86 , srv1720167746:86 , srv1763529663:86 , srv178471067:86 , srv1851825723:86 , srv1975484373:86 , srv2056624196:86 , srv2059539672:86 , srv2073804675:86 , srv2123482192:86 , srv744188861:86 , srv785830970:86 , srv802650834:86 , srv1152816150:87 , srv1770625145:87 , srv1882940550:87 , srv2113368031:87 , srv386886220:87 , srv428340976:87 , srv449257017:87 , srv498271219:87 , srv661307462:87 , srv762387551:87 , srv852961082:87 , srv87102211:87 , srv1106284851:88 , srv129123913:88 , srv1605058400:88 , srv1683286740:88 , srv1708873842:88 , srv1777274885:88 , srv2014106704:88 , srv1444612878:89 , srv1445478585:89 , srv1497672333:89 , srv1522976858:89 , srv1141876085:560 }
2023-10-24 16:53:28,595 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-24 16:53:28,596 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1752215766=45, srv1029052626=3, srv1170626615=11, srv1882940550=55, srv1715591884=42, srv1678834067=37, srv827433754=94, srv1790815033=52, srv386886220=73, srv2041530587=63, srv1247905138=14, srv1463507349=26, srv1773600044=48, srv2073804675=66, srv7476667=88, srv396977076=74, srv1108890087=7, srv1561405676=31, srv653696144=83, srv1152816150=10, srv296343021=72, srv873571285=97, srv1371670363=19, srv1445478585=25, srv1576034011=32, srv695019410=85, srv1407800120=21, srv825836484=93, srv1477690784=27, srv1680449534=38, srv425591188=75, srv1241062912=13, srv636641587=82, srv762387551=89, srv911212690=98, srv1206004521=12, srv1645724857=35, srv916248920=99, srv1434181857=23, srv1686543677=40, srv1522976858=29, srv1719022876=43, srv178471067=50, srv1444612878=24, srv1578499947=33, srv1991560165=60, srv139110028=20, srv1975484373=58, srv744188861=87, srv802650834=92, srv129123913=16, srv2123482192=69, srv193196115=57, srv1899775889=56, srv626721289=81, srv285612114=71, srv444438843=77, srv1106284851=6, srv210458165=67, srv1605058400=34, srv661307462=84, srv852961082=95, srv1016501949=2, srv498271219=79, srv2059539672=65, srv1036020276=4, srv1497672333=28, srv1085717676=5, srv2022456489=62, srv1665593950=36, srv2113368031=68, srv87102211=96, srv2014106704=61, srv1866803148=54, srv1316972006=17, srv233427243=70, srv1141876085=9, srv1000791352=0, srv713027587=86, srv1003346276=1, srv2056624196=64, srv1532585432=30, srv1851825723=53, srv1720167746=44, srv449257017=78, srv1789669963=51, srv1683286740=39, srv428340976=76, srv1763529663=46, srv582636097=80, srv1770625145=47, srv781938911=90, srv1135265785=8, srv785830970=91, srv1350739399=18, srv1777274885=49, srv1708873842=41, srv1409989776=22, srv198653677=59, srv1287600987=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-24 16:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-24 16:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-24 16:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-24 16:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-24 16:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-24 16:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-24 16:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-24 16:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-24 16:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-24 16:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-24 16:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-24 16:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-24 16:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-24 16:53:28,615 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-24 16:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-24 16:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-24 16:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-24 16:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-24 16:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-24 16:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-24 16:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-24 16:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-24 16:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-24 16:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-24 16:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-24 16:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-24 16:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-24 16:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-24 16:53:28,616 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-24 16:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-24 16:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-24 16:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-24 16:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-24 16:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-24 16:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-24 16:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-24 16:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-24 16:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-24 16:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-24 16:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-24 16:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-24 16:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-24 16:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-24 16:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-24 16:53:28,617 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-24 16:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-24 16:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-24 16:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-24 16:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-24 16:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-24 16:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-24 16:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-24 16:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-24 16:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-24 16:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-24 16:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-24 16:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-24 16:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-24 16:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-24 16:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-24 16:53:28,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-24 16:53:28,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-24 16:53:28,620 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-24 16:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-24 16:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-24 16:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-24 16:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-24 16:53:28,621 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-24 16:53:28,732 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0044444444444444444); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00670025210172808); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0044444444444444444); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-24 16:53:28,732 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-24 16:53:28,732 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.742581317042097E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0044444444444444444); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00670025210172808); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0044444444444444444); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=3.333333333333333E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0035833333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-24 16:53:28,950 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 316 ms to try 5000 different iterations.  Found a solution that moves 229 regions; Going from a computed imbalance of 6.742581317042097E-4 to a new imbalance of 1.1104481204574482E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.025444444444444443, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-24 16:53:28,950 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1106284851:82 , srv1522976858:82 , srv1000791352:83 , srv1016501949:83 , srv1497672333:83 , srv1708873842:83 , srv695019410:83 , srv713027587:83 , srv1085717676:84 , srv1135265785:84 , srv1152816150:84 , srv1241062912:84 , srv1247905138:84 , srv129123913:84 , srv1316972006:84 , srv1350739399:84 , srv1409989776:84 , srv1444612878:84 , srv1445478585:84 , srv1532585432:84 , srv1578499947:84 , srv1605058400:84 , srv1645724857:84 , srv1665593950:84 , srv1680449534:84 , srv1686543677:84 , srv1715591884:84 , srv1719022876:84 , srv1752215766:84 , srv1763529663:84 , srv1770625145:84 , srv1777274885:84 , srv1866803148:84 , srv1882940550:84 , srv193196115:84 , srv1975484373:84 , srv2014106704:84 , srv2041530587:84 , srv2056624196:84 , srv2073804675:84 , srv2113368031:84 , srv2123482192:84 , srv296343021:84 , srv386886220:84 , srv396977076:84 , srv428340976:84 , srv444438843:84 , srv449257017:84 , srv498271219:84 , srv626721289:84 , srv744188861:84 , srv7476667:84 , srv785830970:84 , srv852961082:84 , srv916248920:84 , srv1003346276:85 , srv1029052626:85 , srv1108890087:85 , srv1170626615:85 , srv1206004521:85 , srv1287600987:85 , srv1371670363:85 , srv139110028:85 , srv1407800120:85 , srv1434181857:85 , srv1463507349:85 , srv1477690784:85 , srv1561405676:85 , srv1576034011:85 , srv1678834067:85 , srv1683286740:85 , srv1720167746:85 , srv1773600044:85 , srv178471067:85 , srv1789669963:85 , srv1790815033:85 , srv1851825723:85 , srv1899775889:85 , srv1991560165:85 , srv2059539672:85 , srv210458165:85 , srv233427243:85 , srv425591188:85 , srv636641587:85 , srv653696144:85 , srv661307462:85 , srv762387551:85 , srv781938911:85 , srv802650834:85 , srv825836484:85 , srv827433754:85 , srv87102211:85 , srv873571285:85 , srv911212690:85 , srv1036020276:86 , srv198653677:86 , srv2022456489:86 , srv285612114:86 , srv582636097:86 , srv1141876085:645 }
2023-10-24 16:53:28,954 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-24 16:53:28,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1752215766=45, srv1029052626=3, srv1170626615=11, srv1882940550=55, srv1715591884=42, srv1678834067=37, srv827433754=94, srv1790815033=52, srv386886220=73, srv2041530587=63, srv1247905138=14, srv1463507349=26, srv1773600044=48, srv2073804675=66, srv7476667=88, srv396977076=74, srv1108890087=7, srv1561405676=31, srv653696144=83, srv1152816150=10, srv296343021=72, srv873571285=97, srv1371670363=19, srv1445478585=25, srv1576034011=32, srv695019410=85, srv1407800120=21, srv825836484=93, srv1477690784=27, srv1680449534=38, srv425591188=75, srv1241062912=13, srv636641587=82, srv762387551=89, srv911212690=98, srv1206004521=12, srv1645724857=35, srv916248920=99, srv1434181857=23, srv1686543677=40, srv1522976858=29, srv1719022876=43, srv178471067=50, srv1444612878=24, srv1578499947=33, srv1991560165=60, srv139110028=20, srv1975484373=58, srv744188861=87, srv802650834=92, srv129123913=16, srv2123482192=69, srv193196115=57, srv1899775889=56, srv626721289=81, srv285612114=71, srv444438843=77, srv1106284851=6, srv210458165=67, srv1605058400=34, srv661307462=84, srv852961082=95, srv1016501949=2, srv498271219=79, srv2059539672=65, srv1036020276=4, srv1497672333=28, srv1085717676=5, srv2022456489=62, srv1665593950=36, srv2113368031=68, srv87102211=96, srv2014106704=61, srv1866803148=54, srv1316972006=17, srv233427243=70, srv1141876085=9, srv1000791352=0, srv713027587=86, srv1003346276=1, srv2056624196=64, srv1532585432=30, srv1851825723=53, srv1720167746=44, srv449257017=78, srv1789669963=51, srv1683286740=39, srv428340976=76, srv1763529663=46, srv582636097=80, srv1770625145=47, srv781938911=90, srv1135265785=8, srv785830970=91, srv1350739399=18, srv1777274885=49, srv1708873842=41, srv1409989776=22, srv198653677=59, srv1287600987=15} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-24 16:53:28,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-24 16:53:28,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-24 16:53:28,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-24 16:53:28,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-24 16:53:28,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-24 16:53:28,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-24 16:53:28,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-24 16:53:28,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-24 16:53:28,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-24 16:53:28,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-24 16:53:28,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-24 16:53:28,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-24 16:53:28,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-24 16:53:28,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-24 16:53:28,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-24 16:53:28,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-24 16:53:28,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-24 16:53:28,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-24 16:53:28,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-24 16:53:28,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-24 16:53:28,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-24 16:53:28,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-24 16:53:28,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-24 16:53:28,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-24 16:53:28,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-24 16:53:28,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-24 16:53:28,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-24 16:53:28,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-24 16:53:28,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-24 16:53:28,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-24 16:53:28,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-24 16:53:28,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-24 16:53:28,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-24 16:53:28,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-24 16:53:28,970 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-24 16:53:28,971 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-24 16:53:28,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-24 16:53:28,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-24 16:53:28,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-24 16:53:28,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-24 16:53:28,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-24 16:53:28,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-24 16:53:28,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-24 16:53:28,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-24 16:53:28,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-24 16:53:28,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-24 16:53:28,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-24 16:53:28,972 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-24 16:53:29,077 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=9.50098893244694E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=3.158529131194229E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017727200782610348); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=3.158529131194229E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-24 16:53:29,077 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1106284851:82 , srv1522976858:82 , srv1000791352:83 , srv1016501949:83 , srv1497672333:83 , srv1708873842:83 , srv695019410:83 , srv713027587:83 , srv1085717676:84 , srv1135265785:84 , srv1152816150:84 , srv1241062912:84 , srv1247905138:84 , srv129123913:84 , srv1316972006:84 , srv1350739399:84 , srv1409989776:84 , srv1444612878:84 , srv1445478585:84 , srv1532585432:84 , srv1578499947:84 , srv1605058400:84 , srv1645724857:84 , srv1665593950:84 , srv1680449534:84 , srv1686543677:84 , srv1715591884:84 , srv1719022876:84 , srv1752215766:84 , srv1763529663:84 , srv1770625145:84 , srv1777274885:84 , srv1866803148:84 , srv1882940550:84 , srv193196115:84 , srv1975484373:84 , srv2014106704:84 , srv2041530587:84 , srv2056624196:84 , srv2073804675:84 , srv2113368031:84 , srv2123482192:84 , srv296343021:84 , srv386886220:84 , srv396977076:84 , srv428340976:84 , srv444438843:84 , srv449257017:84 , srv498271219:84 , srv626721289:84 , srv744188861:84 , srv7476667:84 , srv785830970:84 , srv852961082:84 , srv916248920:84 , srv1003346276:85 , srv1029052626:85 , srv1108890087:85 , srv1170626615:85 , srv1206004521:85 , srv1287600987:85 , srv1371670363:85 , srv139110028:85 , srv1407800120:85 , srv1434181857:85 , srv1463507349:85 , srv1477690784:85 , srv1561405676:85 , srv1576034011:85 , srv1678834067:85 , srv1683286740:85 , srv1720167746:85 , srv1773600044:85 , srv178471067:85 , srv1789669963:85 , srv1790815033:85 , srv1851825723:85 , srv1899775889:85 , srv1991560165:85 , srv2059539672:85 , srv210458165:85 , srv233427243:85 , srv425591188:85 , srv636641587:85 , srv653696144:85 , srv661307462:85 , srv762387551:85 , srv781938911:85 , srv802650834:85 , srv825836484:85 , srv827433754:85 , srv87102211:85 , srv873571285:85 , srv911212690:85 , srv1036020276:86 , srv198653677:86 , srv2022456489:86 , srv285612114:86 , srv582636097:86 , srv1141876085:645 }
2023-10-24 16:53:29,103 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=357 (was 357), ProcessCount=170 (was 170), AvailableMemoryMB=8104 (was 8613)
2023-10-24 16:53:29,113 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=357, ProcessCount=170, AvailableMemoryMB=8103
2023-10-24 16:53:29,114 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-24 16:53:29,115 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-10-24 16:53:29,123 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv110743353:84 , srv1160876742:84 , srv1181307799:84 , srv12213820:84 , srv1458346764:84 , srv1477685185:84 , srv1552422822:84 , srv1633605014:84 , srv1726149373:84 , srv1835808220:84 , srv1969265544:84 , srv1978186059:84 , srv2013357756:84 , srv2036442057:84 , srv2088686199:84 , srv2139557171:84 , srv234544533:84 , srv243607065:84 , srv26238799:84 , srv263814821:84 , srv353003022:84 , srv471840270:84 , srv657718432:84 , srv677783636:84 , srv704652895:84 , srv789329306:84 , srv920796128:84 , srv92254175:84 , srv947717535:84 , srv1415977075:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-10-24 16:53:29,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-24 16:53:29,124 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv263814821=20, srv92254175=28, srv1633605014=8, srv1835808220=10, srv1181307799=2, srv1458346764=5, srv1969265544=11, srv704652895=25, srv12213820=3, srv1415977075=4, srv2036442057=14, srv2088686199=15, srv471840270=22, srv353003022=21, srv1477685185=6, srv2139557171=16, srv26238799=19, srv657718432=23, srv234544533=17, srv2013357756=13, srv947717535=29, srv1160876742=1, srv677783636=24, srv1978186059=12, srv243607065=18, srv110743353=0, srv1552422822=7, srv789329306=26, srv920796128=27, srv1726149373=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-24 16:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-24 16:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-24 16:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-24 16:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-24 16:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-24 16:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-24 16:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-24 16:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-24 16:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-24 16:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-24 16:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-24 16:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-24 16:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-24 16:53:29,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-24 16:53:29,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-10-24 16:53:29,149 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1356060457535417, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-24 16:53:29,149 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912620333486504, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1356060457535417, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-10-24 16:55:29,137 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6163508 different iterations.  Found a solution that moves 1854 regions; Going from a computed imbalance of 0.9912620333486504 to a new imbalance of 4.3273284898463826E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6866666666666666, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-24 16:55:29,143 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv110743353:90 , srv1160876742:90 , srv1181307799:90 , srv12213820:90 , srv1415977075:90 , srv1458346764:90 , srv1477685185:90 , srv1552422822:90 , srv1633605014:90 , srv1726149373:90 , srv1835808220:90 , srv1969265544:90 , srv1978186059:90 , srv2013357756:90 , srv2036442057:90 , srv2088686199:90 , srv2139557171:90 , srv234544533:90 , srv243607065:90 , srv26238799:90 , srv263814821:90 , srv353003022:90 , srv471840270:90 , srv657718432:90 , srv677783636:90 , srv704652895:90 , srv789329306:90 , srv920796128:90 , srv92254175:90 , srv947717535:90 }
2023-10-24 16:55:29,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-24 16:55:29,144 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv263814821=20, srv92254175=28, srv1633605014=8, srv1835808220=10, srv1181307799=2, srv1458346764=5, srv1969265544=11, srv704652895=25, srv12213820=3, srv1415977075=4, srv2036442057=14, srv2088686199=15, srv471840270=22, srv353003022=21, srv1477685185=6, srv2139557171=16, srv26238799=19, srv657718432=23, srv234544533=17, srv2013357756=13, srv947717535=29, srv1160876742=1, srv677783636=24, srv1978186059=12, srv243607065=18, srv110743353=0, srv1552422822=7, srv789329306=26, srv920796128=27, srv1726149373=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-24 16:55:29,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-24 16:55:29,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-24 16:55:29,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-24 16:55:29,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-24 16:55:29,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-24 16:55:29,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-24 16:55:29,148 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-24 16:55:29,149 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-24 16:55:29,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-24 16:55:29,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-24 16:55:29,150 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-10-24 16:55:29,162 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-24 16:55:29,162 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv110743353:90 , srv1160876742:90 , srv1181307799:90 , srv12213820:90 , srv1415977075:90 , srv1458346764:90 , srv1477685185:90 , srv1552422822:90 , srv1633605014:90 , srv1726149373:90 , srv1835808220:90 , srv1969265544:90 , srv1978186059:90 , srv2013357756:90 , srv2036442057:90 , srv2088686199:90 , srv2139557171:90 , srv234544533:90 , srv243607065:90 , srv26238799:90 , srv263814821:90 , srv353003022:90 , srv471840270:90 , srv657718432:90 , srv677783636:90 , srv704652895:90 , srv789329306:90 , srv920796128:90 , srv92254175:90 , srv947717535:90 }
2023-10-24 16:55:29,174 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=189 (was 357), ProcessCount=168 (was 170), AvailableMemoryMB=7950 (was 8103)