2023-08-18 21:56:42,709 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-08-18 21:56:42,940 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-08-18 21:56:43,032 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-08-18 21:56:43,086 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-08-18 21:56:43,092 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-08-18 21:56:43,103 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=217, ProcessCount=168, AvailableMemoryMB=3848
2023-08-18 21:56:43,105 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-08-18 21:56:43,106 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-08-18 21:56:43,243 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1004703431:84 , srv1061239700:84 , srv106842494:84 , srv1083897677:84 , srv1088434503:84 , srv1151760038:84 , srv1152958963:84 , srv1190274269:84 , srv121501466:84 , srv1237703644:84 , srv1289807551:84 , srv1298602446:84 , srv1323500651:84 , srv1352810926:84 , srv1387415537:84 , srv1402308129:84 , srv1455324269:84 , srv1467502261:84 , srv1470238133:84 , srv1531636280:84 , srv1570850730:84 , srv1571420066:84 , srv158617371:84 , srv1591198137:84 , srv1635678436:84 , srv1652582349:84 , srv1675799971:84 , srv1708128902:84 , srv1709031843:84 , srv170924482:84 , srv1724687625:84 , srv1743743103:84 , srv1747219815:84 , srv1759875634:84 , srv1768285432:84 , srv1797362513:84 , srv1817027834:84 , srv1833597797:84 , srv1848549568:84 , srv1860617345:84 , srv186137801:84 , srv1869675977:84 , srv186983155:84 , srv1877330620:84 , srv1879635566:84 , srv1887284334:84 , srv1916804716:84 , srv1926977192:84 , srv194006749:84 , srv1940708365:84 , srv1954993256:84 , srv198793611:84 , srv1998618304:84 , srv2026920607:84 , srv2037256323:84 , srv2053225545:84 , srv206093117:84 , srv2062407117:84 , srv2063818041:84 , srv2133164601:84 , srv2143435192:84 , srv229703546:84 , srv235526793:84 , srv236515701:84 , srv24119256:84 , srv246207435:84 , srv280690038:84 , srv285512146:84 , srv321396128:84 , srv364336440:84 , srv401035081:84 , srv442086210:84 , srv475842871:84 , srv479257417:84 , srv483430377:84 , srv506433604:84 , srv516774062:84 , srv525555253:84 , srv556225947:84 , srv559448279:84 , srv56414106:84 , srv565294502:84 , srv597616864:84 , srv638337195:84 , srv690999287:84 , srv699999114:84 , srv744237250:84 , srv761512316:84 , srv81754757:84 , srv830954297:84 , srv836879795:84 , srv842567203:84 , srv845125763:84 , srv87525254:84 , srv923369943:84 , srv940424601:84 , srv962609252:84 , srv985346738:84 , srv986844148:84 , srv1961424502:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-08-18 21:56:43,256 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-18 21:56:43,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1323500651=12, srv1940708365=49, srv170924482=29, srv1916804716=46, srv1467502261=17, srv1591198137=23, srv280690038=67, srv235526793=63, srv962609252=97, srv1298602446=11, srv1088434503=4, srv1061239700=1, srv1887284334=45, srv940424601=96, srv364336440=70, srv2063818041=59, srv1289807551=10, srv1570850730=20, srv87525254=94, srv830954297=90, srv1848549568=38, srv1860617345=39, srv845125763=93, srv1879635566=44, srv2133164601=60, srv1961424502=51, srv1387415537=14, srv699999114=86, srv1877330620=43, srv986844148=99, srv690999287=85, srv1470238133=18, srv1571420066=21, srv1743743103=31, srv1797362513=35, srv506433604=76, srv81754757=89, srv246207435=66, srv1709031843=28, srv2037256323=55, srv556225947=79, srv479257417=74, srv186983155=42, srv229703546=62, srv1635678436=24, srv1724687625=30, srv24119256=65, srv2143435192=61, srv236515701=64, srv525555253=78, srv985346738=98, srv121501466=8, srv923369943=95, srv638337195=84, srv1237703644=9, srv1954993256=50, srv1151760038=5, srv1675799971=26, srv1747219815=32, srv106842494=2, srv1083897677=3, srv559448279=80, srv1768285432=34, srv761512316=88, srv1152958963=6, srv1998618304=53, srv1926977192=47, srv56414106=81, srv186137801=40, srv2062407117=58, srv1833597797=37, srv198793611=52, srv321396128=69, srv1817027834=36, srv1531636280=19, srv1352810926=13, srv2053225545=56, srv597616864=83, srv565294502=82, srv401035081=71, srv442086210=72, srv516774062=77, srv1869675977=41, srv483430377=75, srv194006749=48, srv475842871=73, srv1455324269=16, srv206093117=57, srv836879795=91, srv1004703431=0, srv2026920607=54, srv158617371=22, srv1652582349=25, srv842567203=92, srv1402308129=15, srv285512146=68, srv1759875634=33, srv744237250=87, srv1708128902=27, srv1190274269=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-18 21:56:43,323 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-18 21:56:43,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-18 21:56:43,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-18 21:56:43,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-18 21:56:43,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-18 21:56:43,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-18 21:56:43,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-18 21:56:43,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-18 21:56:43,324 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-18 21:56:43,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-18 21:56:43,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-18 21:56:43,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-18 21:56:43,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-18 21:56:43,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-18 21:56:43,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-18 21:56:43,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-18 21:56:43,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-18 21:56:43,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-18 21:56:43,325 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-18 21:56:43,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-18 21:56:43,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-18 21:56:43,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-18 21:56:43,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-18 21:56:43,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-18 21:56:43,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-18 21:56:43,326 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-18 21:56:43,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-18 21:56:43,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-18 21:56:43,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-18 21:56:43,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-18 21:56:43,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-18 21:56:43,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-18 21:56:43,327 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-18 21:56:43,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-18 21:56:43,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-18 21:56:43,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-18 21:56:43,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-18 21:56:43,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-18 21:56:43,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-18 21:56:43,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-18 21:56:43,328 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-18 21:56:43,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-18 21:56:43,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-18 21:56:43,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-18 21:56:43,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-18 21:56:43,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-18 21:56:43,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-18 21:56:43,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-18 21:56:43,329 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-18 21:56:43,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-18 21:56:43,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-18 21:56:43,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-18 21:56:43,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-18 21:56:43,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-18 21:56:43,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-18 21:56:43,330 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-18 21:56:43,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-18 21:56:43,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-18 21:56:43,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-18 21:56:43,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-18 21:56:43,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-18 21:56:43,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-18 21:56:43,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-18 21:56:43,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-18 21:56:43,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-18 21:56:43,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-18 21:56:43,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-18 21:56:43,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-18 21:56:43,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-18 21:56:43,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-18 21:56:43,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-18 21:56:43,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-18 21:56:43,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-18 21:56:43,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-18 21:56:43,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-18 21:56:43,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-18 21:56:43,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-18 21:56:43,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-18 21:56:43,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-18 21:56:43,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-18 21:56:43,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-18 21:56:43,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-18 21:56:43,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-18 21:56:43,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-18 21:56:43,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-18 21:56:43,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-18 21:56:43,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-18 21:56:43,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-18 21:56:43,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-18 21:56:43,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-18 21:56:43,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-18 21:56:43,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-18 21:56:43,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-18 21:56:43,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-18 21:56:43,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-18 21:56:43,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-18 21:56:43,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-18 21:56:43,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-18 21:56:43,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-18 21:56:43,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-18 21:56:43,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-18 21:56:43,649 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-18 21:56:43,649 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-18 21:56:43,650 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-18 21:56:43,932 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 566 ms to try 5000 different iterations.  Found a solution that moves 4694 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18139581234573177. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03845598394378855, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05602228192272612, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5215555555555556, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03845598394378855, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17675, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24216666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-18 21:56:43,947 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1848549568:81 , srv158617371:82 , srv1724687625:83 , srv2062407117:83 , srv525555253:83 , srv81754757:83 , srv1004703431:84 , srv1151760038:84 , srv1190274269:84 , srv1352810926:84 , srv1652582349:84 , srv1747219815:84 , srv1879635566:84 , srv1887284334:84 , srv1954993256:84 , srv235526793:84 , srv321396128:84 , srv556225947:84 , srv690999287:84 , srv836879795:84 , srv1470238133:85 , srv1571420066:85 , srv194006749:85 , srv198793611:85 , srv2026920607:85 , srv206093117:85 , srv2063818041:85 , srv280690038:85 , srv761512316:85 , srv940424601:85 , srv1237703644:86 , srv1402308129:86 , srv1591198137:86 , srv1768285432:86 , srv1860617345:86 , srv1998618304:86 , srv2037256323:86 , srv229703546:86 , srv246207435:86 , srv479257417:86 , srv506433604:86 , srv516774062:86 , srv638337195:86 , srv923369943:86 , srv1061239700:87 , srv121501466:87 , srv1298602446:87 , srv1455324269:87 , srv1531636280:87 , srv1570850730:87 , srv1635678436:87 , srv1675799971:87 , srv170924482:87 , srv1759875634:87 , srv1869675977:87 , srv186983155:87 , srv1877330620:87 , srv1926977192:87 , srv2053225545:87 , srv236515701:87 , srv24119256:87 , srv364336440:87 , srv442086210:87 , srv56414106:87 , srv565294502:87 , srv699999114:87 , srv744237250:87 , srv842567203:87 , srv845125763:87 , srv985346738:87 , srv106842494:88 , srv1088434503:88 , srv1152958963:88 , srv1289807551:88 , srv1387415537:88 , srv1467502261:88 , srv1708128902:88 , srv1709031843:88 , srv186137801:88 , srv1916804716:88 , srv2133164601:88 , srv285512146:88 , srv401035081:88 , srv830954297:88 , srv1323500651:89 , srv1743743103:89 , srv1940708365:89 , srv2143435192:89 , srv87525254:89 , srv986844148:89 , srv1083897677:90 , srv1817027834:90 , srv1833597797:90 , srv483430377:90 , srv597616864:90 , srv962609252:90 , srv1797362513:91 , srv475842871:91 , srv559448279:93 , srv1961424502:432 }
2023-08-18 21:56:43,953 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-18 21:56:43,955 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1323500651=12, srv1940708365=49, srv170924482=29, srv1916804716=46, srv1467502261=17, srv1591198137=23, srv280690038=67, srv235526793=63, srv962609252=97, srv1298602446=11, srv1088434503=4, srv1061239700=1, srv1887284334=45, srv940424601=96, srv364336440=70, srv2063818041=59, srv1289807551=10, srv1570850730=20, srv87525254=94, srv830954297=90, srv1848549568=38, srv1860617345=39, srv845125763=93, srv1879635566=44, srv2133164601=60, srv1961424502=51, srv1387415537=14, srv699999114=86, srv1877330620=43, srv986844148=99, srv690999287=85, srv1470238133=18, srv1571420066=21, srv1743743103=31, srv1797362513=35, srv506433604=76, srv81754757=89, srv246207435=66, srv1709031843=28, srv2037256323=55, srv556225947=79, srv479257417=74, srv186983155=42, srv229703546=62, srv1635678436=24, srv1724687625=30, srv24119256=65, srv2143435192=61, srv236515701=64, srv525555253=78, srv985346738=98, srv121501466=8, srv923369943=95, srv638337195=84, srv1237703644=9, srv1954993256=50, srv1151760038=5, srv1675799971=26, srv1747219815=32, srv106842494=2, srv1083897677=3, srv559448279=80, srv1768285432=34, srv761512316=88, srv1152958963=6, srv1998618304=53, srv1926977192=47, srv56414106=81, srv186137801=40, srv2062407117=58, srv1833597797=37, srv198793611=52, srv321396128=69, srv1817027834=36, srv1531636280=19, srv1352810926=13, srv2053225545=56, srv597616864=83, srv565294502=82, srv401035081=71, srv442086210=72, srv516774062=77, srv1869675977=41, srv483430377=75, srv194006749=48, srv475842871=73, srv1455324269=16, srv206093117=57, srv836879795=91, srv1004703431=0, srv2026920607=54, srv158617371=22, srv1652582349=25, srv842567203=92, srv1402308129=15, srv285512146=68, srv1759875634=33, srv744237250=87, srv1708128902=27, srv1190274269=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-18 21:56:43,978 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-18 21:56:43,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-18 21:56:43,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-18 21:56:43,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-18 21:56:43,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-18 21:56:43,979 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-18 21:56:43,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-18 21:56:43,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-18 21:56:43,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-18 21:56:43,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-18 21:56:43,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-18 21:56:43,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-18 21:56:43,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-18 21:56:43,980 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-18 21:56:43,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-18 21:56:43,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-18 21:56:43,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-18 21:56:43,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-18 21:56:43,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-18 21:56:43,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-18 21:56:43,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-18 21:56:43,981 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-18 21:56:43,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-18 21:56:43,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-18 21:56:43,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-18 21:56:43,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-18 21:56:43,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-18 21:56:43,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-18 21:56:43,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-18 21:56:43,982 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-18 21:56:43,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-18 21:56:43,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-18 21:56:43,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-18 21:56:43,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-18 21:56:43,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-18 21:56:43,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-18 21:56:43,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-18 21:56:43,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-18 21:56:43,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-18 21:56:43,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-18 21:56:43,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-18 21:56:43,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-18 21:56:43,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-18 21:56:43,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-18 21:56:43,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-18 21:56:43,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-18 21:56:43,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-18 21:56:43,984 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-18 21:56:43,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-18 21:56:43,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-18 21:56:43,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-18 21:56:43,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-18 21:56:43,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-18 21:56:43,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-18 21:56:43,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-18 21:56:43,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-18 21:56:43,985 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-18 21:56:43,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-18 21:56:43,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-18 21:56:43,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-18 21:56:43,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-18 21:56:43,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-18 21:56:43,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-18 21:56:43,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-18 21:56:43,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-18 21:56:43,986 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-18 21:56:43,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-18 21:56:43,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-18 21:56:43,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-18 21:56:43,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-18 21:56:43,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-18 21:56:43,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-18 21:56:43,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-18 21:56:43,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-18 21:56:43,987 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-18 21:56:43,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-18 21:56:43,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-18 21:56:43,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-18 21:56:43,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-18 21:56:43,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-18 21:56:43,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-18 21:56:43,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-18 21:56:43,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-18 21:56:43,988 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-18 21:56:43,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-18 21:56:43,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-18 21:56:43,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-18 21:56:43,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-18 21:56:43,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-18 21:56:43,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-18 21:56:43,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-18 21:56:43,989 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-18 21:56:43,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-18 21:56:43,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-18 21:56:43,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-18 21:56:43,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-18 21:56:43,990 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-18 21:56:43,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-18 21:56:43,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-18 21:56:43,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-18 21:56:43,991 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-18 21:56:44,227 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03845598394378855, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05602228192272612, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03845598394378855, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17675, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24216666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-18 21:56:44,228 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-18 21:56:44,228 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18136294425522798, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03845598394378855, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05602228192272612, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03845598394378855, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17675, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24216666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-18 21:56:44,479 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 461 ms to try 5000 different iterations.  Found a solution that moves 2582 regions; Going from a computed imbalance of 0.18136294425522798 to a new imbalance of 0.01459122609756577. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022175596428106778); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03375490985610744, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2868888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022175596428106778); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042333333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-18 21:56:44,491 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv559448279:80 , srv475842871:81 , srv483430377:81 , srv106842494:82 , srv1323500651:82 , srv1759875634:82 , srv1797362513:82 , srv1940708365:82 , srv2053225545:82 , srv87525254:82 , srv986844148:82 , srv1083897677:83 , srv1088434503:83 , srv1387415537:83 , srv1455324269:83 , srv1467502261:83 , srv1570850730:83 , srv1675799971:83 , srv1708128902:83 , srv170924482:83 , srv1743743103:83 , srv1833597797:83 , srv186137801:83 , srv186983155:83 , srv2063818041:83 , srv56414106:83 , srv597616864:83 , srv962609252:83 , srv1237703644:84 , srv1402308129:84 , srv1470238133:84 , srv1709031843:84 , srv1817027834:84 , srv1998618304:84 , srv2133164601:84 , srv699999114:84 , srv744237250:84 , srv845125763:84 , srv923369943:84 , srv1152958963:85 , srv121501466:85 , srv1298602446:85 , srv1635678436:85 , srv1877330620:85 , srv236515701:85 , srv24119256:85 , srv285512146:85 , srv401035081:85 , srv565294502:85 , srv830954297:85 , srv842567203:85 , srv1061239700:86 , srv1591198137:86 , srv1916804716:86 , srv1954993256:86 , srv2026920607:86 , srv2037256323:86 , srv2143435192:86 , srv364336440:86 , srv479257417:86 , srv506433604:86 , srv940424601:86 , srv985346738:86 , srv1289807551:87 , srv1531636280:87 , srv1571420066:87 , srv1724687625:87 , srv1887284334:87 , srv1926977192:87 , srv246207435:87 , srv321396128:87 , srv525555253:87 , srv638337195:87 , srv761512316:87 , srv1151760038:88 , srv1190274269:88 , srv1747219815:88 , srv1768285432:88 , srv1860617345:88 , srv1869675977:88 , srv206093117:88 , srv229703546:88 , srv235526793:88 , srv280690038:88 , srv442086210:88 , srv516774062:88 , srv556225947:88 , srv690999287:88 , srv836879795:88 , srv1004703431:89 , srv158617371:89 , srv1652582349:89 , srv194006749:89 , srv198793611:89 , srv1352810926:90 , srv1879635566:90 , srv1848549568:91 , srv81754757:91 , srv2062407117:92 , srv1961424502:539 }
2023-08-18 21:56:44,497 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-18 21:56:44,498 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1323500651=12, srv1940708365=49, srv170924482=29, srv1916804716=46, srv1467502261=17, srv1591198137=23, srv280690038=67, srv235526793=63, srv962609252=97, srv1298602446=11, srv1088434503=4, srv1061239700=1, srv1887284334=45, srv940424601=96, srv364336440=70, srv2063818041=59, srv1289807551=10, srv1570850730=20, srv87525254=94, srv830954297=90, srv1848549568=38, srv1860617345=39, srv845125763=93, srv1879635566=44, srv2133164601=60, srv1961424502=51, srv1387415537=14, srv699999114=86, srv1877330620=43, srv986844148=99, srv690999287=85, srv1470238133=18, srv1571420066=21, srv1743743103=31, srv1797362513=35, srv506433604=76, srv81754757=89, srv246207435=66, srv1709031843=28, srv2037256323=55, srv556225947=79, srv479257417=74, srv186983155=42, srv229703546=62, srv1635678436=24, srv1724687625=30, srv24119256=65, srv2143435192=61, srv236515701=64, srv525555253=78, srv985346738=98, srv121501466=8, srv923369943=95, srv638337195=84, srv1237703644=9, srv1954993256=50, srv1151760038=5, srv1675799971=26, srv1747219815=32, srv106842494=2, srv1083897677=3, srv559448279=80, srv1768285432=34, srv761512316=88, srv1152958963=6, srv1998618304=53, srv1926977192=47, srv56414106=81, srv186137801=40, srv2062407117=58, srv1833597797=37, srv198793611=52, srv321396128=69, srv1817027834=36, srv1531636280=19, srv1352810926=13, srv2053225545=56, srv597616864=83, srv565294502=82, srv401035081=71, srv442086210=72, srv516774062=77, srv1869675977=41, srv483430377=75, srv194006749=48, srv475842871=73, srv1455324269=16, srv206093117=57, srv836879795=91, srv1004703431=0, srv2026920607=54, srv158617371=22, srv1652582349=25, srv842567203=92, srv1402308129=15, srv285512146=68, srv1759875634=33, srv744237250=87, srv1708128902=27, srv1190274269=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-18 21:56:44,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-18 21:56:44,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-18 21:56:44,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-18 21:56:44,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-18 21:56:44,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-18 21:56:44,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-18 21:56:44,517 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-18 21:56:44,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-18 21:56:44,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-18 21:56:44,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-18 21:56:44,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-18 21:56:44,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-18 21:56:44,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-18 21:56:44,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-18 21:56:44,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-18 21:56:44,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-18 21:56:44,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-18 21:56:44,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-18 21:56:44,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-18 21:56:44,518 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-18 21:56:44,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-18 21:56:44,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-18 21:56:44,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-18 21:56:44,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-18 21:56:44,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-18 21:56:44,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-18 21:56:44,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-18 21:56:44,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-18 21:56:44,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-18 21:56:44,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-18 21:56:44,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-18 21:56:44,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-18 21:56:44,519 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-18 21:56:44,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-18 21:56:44,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-18 21:56:44,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-18 21:56:44,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-18 21:56:44,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-18 21:56:44,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-18 21:56:44,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-18 21:56:44,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-18 21:56:44,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-18 21:56:44,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-18 21:56:44,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-18 21:56:44,520 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-18 21:56:44,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-18 21:56:44,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-18 21:56:44,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-18 21:56:44,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-18 21:56:44,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-18 21:56:44,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-18 21:56:44,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-18 21:56:44,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-18 21:56:44,521 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-18 21:56:44,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-18 21:56:44,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-18 21:56:44,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-18 21:56:44,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-18 21:56:44,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-18 21:56:44,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-18 21:56:44,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-18 21:56:44,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-18 21:56:44,522 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-18 21:56:44,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-18 21:56:44,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-18 21:56:44,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-18 21:56:44,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-18 21:56:44,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-18 21:56:44,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-18 21:56:44,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-18 21:56:44,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-18 21:56:44,523 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-18 21:56:44,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-18 21:56:44,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-18 21:56:44,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-18 21:56:44,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-18 21:56:44,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-18 21:56:44,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-18 21:56:44,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-18 21:56:44,524 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-18 21:56:44,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-18 21:56:44,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-18 21:56:44,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-18 21:56:44,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-18 21:56:44,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-18 21:56:44,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-18 21:56:44,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-18 21:56:44,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-18 21:56:44,525 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-18 21:56:44,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-18 21:56:44,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-18 21:56:44,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-18 21:56:44,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-18 21:56:44,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-18 21:56:44,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-18 21:56:44,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-18 21:56:44,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-18 21:56:44,526 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-18 21:56:44,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-18 21:56:44,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-18 21:56:44,527 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-18 21:56:44,671 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022175596428106778); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03375490985610744, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022175596428106778); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042333333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-18 21:56:44,671 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-18 21:56:44,671 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.01457314654714379, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022175596428106778); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03375490985610744, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022175596428106778); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011666666666666667); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.042333333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-18 21:56:44,895 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 343 ms to try 5000 different iterations.  Found a solution that moves 941 regions; Going from a computed imbalance of 0.01457314654714379 to a new imbalance of 8.607837778637171E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006657307309718915); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0076394627883592066); MoveCostFunction : (multiplier=7.0, imbalance=0.10455555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006657307309718915); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-18 21:56:44,897 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv442086210:82 , srv1289807551:83 , srv1768285432:83 , srv1817027834:83 , srv1860617345:83 , srv194006749:83 , srv2062407117:83 , srv2143435192:83 , srv516774062:83 , srv962609252:83 , srv1061239700:84 , srv1083897677:84 , srv1152958963:84 , srv1352810926:84 , srv1531636280:84 , srv1591198137:84 , srv1652582349:84 , srv1797362513:84 , srv1833597797:84 , srv1869675977:84 , srv1916804716:84 , srv1926977192:84 , srv198793611:84 , srv206093117:84 , srv229703546:84 , srv280690038:84 , srv401035081:84 , srv597616864:84 , srv638337195:84 , srv1004703431:85 , srv1151760038:85 , srv1190274269:85 , srv1298602446:85 , srv1571420066:85 , srv1635678436:85 , srv1709031843:85 , srv1747219815:85 , srv1877330620:85 , srv1879635566:85 , srv2037256323:85 , srv236515701:85 , srv246207435:85 , srv285512146:85 , srv364336440:85 , srv475842871:85 , srv479257417:85 , srv506433604:85 , srv556225947:85 , srv559448279:85 , srv690999287:85 , srv761512316:85 , srv81754757:85 , srv830954297:85 , srv836879795:85 , srv985346738:85 , srv1088434503:86 , srv121501466:86 , srv1323500651:86 , srv1387415537:86 , srv1708128902:86 , srv1743743103:86 , srv186137801:86 , srv1940708365:86 , srv2133164601:86 , srv24119256:86 , srv321396128:86 , srv483430377:86 , srv565294502:86 , srv744237250:86 , srv842567203:86 , srv87525254:86 , srv940424601:86 , srv986844148:86 , srv106842494:87 , srv1237703644:87 , srv1402308129:87 , srv1467502261:87 , srv1570850730:87 , srv158617371:87 , srv1675799971:87 , srv170924482:87 , srv1724687625:87 , srv1848549568:87 , srv1887284334:87 , srv1954993256:87 , srv1998618304:87 , srv2026920607:87 , srv235526793:87 , srv56414106:87 , srv699999114:87 , srv845125763:87 , srv923369943:87 , srv1455324269:88 , srv1759875634:88 , srv186983155:88 , srv2053225545:88 , srv525555253:88 , srv1470238133:89 , srv2063818041:89 , srv1961424502:546 }
2023-08-18 21:56:44,901 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-18 21:56:44,902 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1323500651=12, srv1940708365=49, srv170924482=29, srv1916804716=46, srv1467502261=17, srv1591198137=23, srv280690038=67, srv235526793=63, srv962609252=97, srv1298602446=11, srv1088434503=4, srv1061239700=1, srv1887284334=45, srv940424601=96, srv364336440=70, srv2063818041=59, srv1289807551=10, srv1570850730=20, srv87525254=94, srv830954297=90, srv1848549568=38, srv1860617345=39, srv845125763=93, srv1879635566=44, srv2133164601=60, srv1961424502=51, srv1387415537=14, srv699999114=86, srv1877330620=43, srv986844148=99, srv690999287=85, srv1470238133=18, srv1571420066=21, srv1743743103=31, srv1797362513=35, srv506433604=76, srv81754757=89, srv246207435=66, srv1709031843=28, srv2037256323=55, srv556225947=79, srv479257417=74, srv186983155=42, srv229703546=62, srv1635678436=24, srv1724687625=30, srv24119256=65, srv2143435192=61, srv236515701=64, srv525555253=78, srv985346738=98, srv121501466=8, srv923369943=95, srv638337195=84, srv1237703644=9, srv1954993256=50, srv1151760038=5, srv1675799971=26, srv1747219815=32, srv106842494=2, srv1083897677=3, srv559448279=80, srv1768285432=34, srv761512316=88, srv1152958963=6, srv1998618304=53, srv1926977192=47, srv56414106=81, srv186137801=40, srv2062407117=58, srv1833597797=37, srv198793611=52, srv321396128=69, srv1817027834=36, srv1531636280=19, srv1352810926=13, srv2053225545=56, srv597616864=83, srv565294502=82, srv401035081=71, srv442086210=72, srv516774062=77, srv1869675977=41, srv483430377=75, srv194006749=48, srv475842871=73, srv1455324269=16, srv206093117=57, srv836879795=91, srv1004703431=0, srv2026920607=54, srv158617371=22, srv1652582349=25, srv842567203=92, srv1402308129=15, srv285512146=68, srv1759875634=33, srv744237250=87, srv1708128902=27, srv1190274269=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-18 21:56:44,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-18 21:56:44,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-18 21:56:44,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-18 21:56:44,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-18 21:56:44,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-18 21:56:44,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-18 21:56:44,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-18 21:56:44,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-18 21:56:44,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-18 21:56:44,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-18 21:56:44,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-18 21:56:44,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-18 21:56:44,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-18 21:56:44,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-18 21:56:44,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-18 21:56:44,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-18 21:56:44,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-18 21:56:44,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-18 21:56:44,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-18 21:56:44,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-18 21:56:44,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-18 21:56:44,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-18 21:56:44,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-18 21:56:44,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-18 21:56:44,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-18 21:56:44,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-18 21:56:44,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-18 21:56:44,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-18 21:56:44,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-18 21:56:44,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-18 21:56:44,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-18 21:56:44,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-18 21:56:44,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-18 21:56:44,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-18 21:56:44,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-18 21:56:44,919 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-18 21:56:45,024 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006657307309718915); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0076394627883592066); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006657307309718915); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-18 21:56:45,024 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-08-18 21:56:45,024 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=8.541947550337085E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006657307309718915); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0076394627883592066); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006657307309718915); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=5.0E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.00375); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-08-18 21:56:45,260 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 331 ms to try 5000 different iterations.  Found a solution that moves 248 regions; Going from a computed imbalance of 8.541947550337085E-4 to a new imbalance of 1.121515704119602E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.027555555555555555, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-18 21:56:45,260 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1151760038:83 , srv1531636280:83 , srv1848549568:83 , srv1869675977:83 , srv235526793:83 , srv81754757:83 , srv121501466:84 , srv1289807551:84 , srv1352810926:84 , srv1455324269:84 , srv1467502261:84 , srv1470238133:84 , srv1708128902:84 , srv1743743103:84 , srv1747219815:84 , srv1759875634:84 , srv1879635566:84 , srv1887284334:84 , srv1916804716:84 , srv1926977192:84 , srv198793611:84 , srv2026920607:84 , srv2037256323:84 , srv2062407117:84 , srv2133164601:84 , srv2143435192:84 , srv229703546:84 , srv24119256:84 , srv246207435:84 , srv285512146:84 , srv364336440:84 , srv559448279:84 , srv565294502:84 , srv744237250:84 , srv830954297:84 , srv842567203:84 , srv845125763:84 , srv986844148:84 , srv1004703431:85 , srv1061239700:85 , srv106842494:85 , srv1083897677:85 , srv1088434503:85 , srv1152958963:85 , srv1190274269:85 , srv1237703644:85 , srv1298602446:85 , srv1387415537:85 , srv1402308129:85 , srv1570850730:85 , srv1571420066:85 , srv158617371:85 , srv1635678436:85 , srv1652582349:85 , srv1675799971:85 , srv1709031843:85 , srv170924482:85 , srv1724687625:85 , srv1768285432:85 , srv1797362513:85 , srv1833597797:85 , srv1860617345:85 , srv186137801:85 , srv186983155:85 , srv1877330620:85 , srv194006749:85 , srv1940708365:85 , srv1954993256:85 , srv1998618304:85 , srv2053225545:85 , srv206093117:85 , srv2063818041:85 , srv236515701:85 , srv280690038:85 , srv321396128:85 , srv401035081:85 , srv442086210:85 , srv475842871:85 , srv479257417:85 , srv483430377:85 , srv506433604:85 , srv516774062:85 , srv525555253:85 , srv556225947:85 , srv56414106:85 , srv597616864:85 , srv638337195:85 , srv690999287:85 , srv699999114:85 , srv761512316:85 , srv836879795:85 , srv87525254:85 , srv923369943:85 , srv940424601:85 , srv985346738:85 , srv1323500651:86 , srv1591198137:86 , srv1817027834:86 , srv962609252:86 , srv1961424502:625 }
2023-08-18 21:56:45,263 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-18 21:56:45,264 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1323500651=12, srv1940708365=49, srv170924482=29, srv1916804716=46, srv1467502261=17, srv1591198137=23, srv280690038=67, srv235526793=63, srv962609252=97, srv1298602446=11, srv1088434503=4, srv1061239700=1, srv1887284334=45, srv940424601=96, srv364336440=70, srv2063818041=59, srv1289807551=10, srv1570850730=20, srv87525254=94, srv830954297=90, srv1848549568=38, srv1860617345=39, srv845125763=93, srv1879635566=44, srv2133164601=60, srv1961424502=51, srv1387415537=14, srv699999114=86, srv1877330620=43, srv986844148=99, srv690999287=85, srv1470238133=18, srv1571420066=21, srv1743743103=31, srv1797362513=35, srv506433604=76, srv81754757=89, srv246207435=66, srv1709031843=28, srv2037256323=55, srv556225947=79, srv479257417=74, srv186983155=42, srv229703546=62, srv1635678436=24, srv1724687625=30, srv24119256=65, srv2143435192=61, srv236515701=64, srv525555253=78, srv985346738=98, srv121501466=8, srv923369943=95, srv638337195=84, srv1237703644=9, srv1954993256=50, srv1151760038=5, srv1675799971=26, srv1747219815=32, srv106842494=2, srv1083897677=3, srv559448279=80, srv1768285432=34, srv761512316=88, srv1152958963=6, srv1998618304=53, srv1926977192=47, srv56414106=81, srv186137801=40, srv2062407117=58, srv1833597797=37, srv198793611=52, srv321396128=69, srv1817027834=36, srv1531636280=19, srv1352810926=13, srv2053225545=56, srv597616864=83, srv565294502=82, srv401035081=71, srv442086210=72, srv516774062=77, srv1869675977=41, srv483430377=75, srv194006749=48, srv475842871=73, srv1455324269=16, srv206093117=57, srv836879795=91, srv1004703431=0, srv2026920607=54, srv158617371=22, srv1652582349=25, srv842567203=92, srv1402308129=15, srv285512146=68, srv1759875634=33, srv744237250=87, srv1708128902=27, srv1190274269=7} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-18 21:56:45,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-18 21:56:45,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-18 21:56:45,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-18 21:56:45,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-18 21:56:45,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-18 21:56:45,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-18 21:56:45,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-18 21:56:45,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-18 21:56:45,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-18 21:56:45,273 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-18 21:56:45,274 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-08-18 21:56:45,275 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-08-18 21:56:45,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-08-18 21:56:45,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-08-18 21:56:45,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-08-18 21:56:45,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-08-18 21:56:45,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-08-18 21:56:45,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-08-18 21:56:45,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-08-18 21:56:45,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-08-18 21:56:45,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-08-18 21:56:45,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-08-18 21:56:45,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-08-18 21:56:45,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-08-18 21:56:45,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-08-18 21:56:45,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-08-18 21:56:45,276 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-08-18 21:56:45,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-08-18 21:56:45,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-08-18 21:56:45,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-08-18 21:56:45,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-08-18 21:56:45,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-08-18 21:56:45,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-08-18 21:56:45,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-08-18 21:56:45,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-08-18 21:56:45,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-08-18 21:56:45,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-08-18 21:56:45,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-08-18 21:56:45,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-08-18 21:56:45,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-08-18 21:56:45,277 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-08-18 21:56:45,278 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-08-18 21:56:45,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-08-18 21:56:45,279 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-08-18 21:56:45,361 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=9.478623925529509E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.994073055283275E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0015713484026367722); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.994073055283275E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-18 21:56:45,361 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1151760038:83 , srv1531636280:83 , srv1848549568:83 , srv1869675977:83 , srv235526793:83 , srv81754757:83 , srv121501466:84 , srv1289807551:84 , srv1352810926:84 , srv1455324269:84 , srv1467502261:84 , srv1470238133:84 , srv1708128902:84 , srv1743743103:84 , srv1747219815:84 , srv1759875634:84 , srv1879635566:84 , srv1887284334:84 , srv1916804716:84 , srv1926977192:84 , srv198793611:84 , srv2026920607:84 , srv2037256323:84 , srv2062407117:84 , srv2133164601:84 , srv2143435192:84 , srv229703546:84 , srv24119256:84 , srv246207435:84 , srv285512146:84 , srv364336440:84 , srv559448279:84 , srv565294502:84 , srv744237250:84 , srv830954297:84 , srv842567203:84 , srv845125763:84 , srv986844148:84 , srv1004703431:85 , srv1061239700:85 , srv106842494:85 , srv1083897677:85 , srv1088434503:85 , srv1152958963:85 , srv1190274269:85 , srv1237703644:85 , srv1298602446:85 , srv1387415537:85 , srv1402308129:85 , srv1570850730:85 , srv1571420066:85 , srv158617371:85 , srv1635678436:85 , srv1652582349:85 , srv1675799971:85 , srv1709031843:85 , srv170924482:85 , srv1724687625:85 , srv1768285432:85 , srv1797362513:85 , srv1833597797:85 , srv1860617345:85 , srv186137801:85 , srv186983155:85 , srv1877330620:85 , srv194006749:85 , srv1940708365:85 , srv1954993256:85 , srv1998618304:85 , srv2053225545:85 , srv206093117:85 , srv2063818041:85 , srv236515701:85 , srv280690038:85 , srv321396128:85 , srv401035081:85 , srv442086210:85 , srv475842871:85 , srv479257417:85 , srv483430377:85 , srv506433604:85 , srv516774062:85 , srv525555253:85 , srv556225947:85 , srv56414106:85 , srv597616864:85 , srv638337195:85 , srv690999287:85 , srv699999114:85 , srv761512316:85 , srv836879795:85 , srv87525254:85 , srv923369943:85 , srv940424601:85 , srv985346738:85 , srv1323500651:86 , srv1591198137:86 , srv1817027834:86 , srv962609252:86 , srv1961424502:625 }
2023-08-18 21:56:45,380 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=240 (was 217) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=3349 (was 3848)
2023-08-18 21:56:45,388 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=240, ProcessCount=168, AvailableMemoryMB=3349
2023-08-18 21:56:45,389 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-08-18 21:56:45,389 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-08-18 21:56:45,395 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1009127412:84 , srv1024701060:84 , srv1033478909:84 , srv1144184729:84 , srv1164087204:84 , srv1180414629:84 , srv1226355896:84 , srv1370481224:84 , srv1471198100:84 , srv1784555884:84 , srv1843890911:84 , srv1917423045:84 , srv1917449428:84 , srv1956288146:84 , srv1957196110:84 , srv1993831249:84 , srv2053379455:84 , srv213623998:84 , srv233243049:84 , srv423844926:84 , srv446805495:84 , srv573148990:84 , srv591512608:84 , srv603506708:84 , srv728970617:84 , srv850174758:84 , srv877305018:84 , srv885192559:84 , srv949066951:84 , srv786541438:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-08-18 21:56:45,396 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-18 21:56:45,397 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2053379455=16, srv233243049=18, srv1009127412=0, srv1784555884=9, srv591512608=22, srv573148990=21, srv1843890911=10, srv1024701060=1, srv885192559=28, srv1917449428=12, srv1164087204=4, srv1226355896=6, srv1956288146=13, srv728970617=24, srv1144184729=3, srv1033478909=2, srv1370481224=7, srv1957196110=14, srv423844926=19, srv1180414629=5, srv213623998=17, srv786541438=25, srv446805495=20, srv1471198100=8, srv1917423045=11, srv877305018=27, srv603506708=23, srv850174758=26, srv1993831249=15, srv949066951=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-18 21:56:45,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-18 21:56:45,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-18 21:56:45,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-18 21:56:45,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-18 21:56:45,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-18 21:56:45,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-18 21:56:45,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-18 21:56:45,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-18 21:56:45,404 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-18 21:56:45,405 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-18 21:56:45,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-18 21:56:45,406 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-08-18 21:56:45,421 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0898391613265412, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-18 21:56:45,421 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912476123528101, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0898391613265412, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-08-18 21:58:45,408 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9426994 different iterations.  Found a solution that moves 1835 regions; Going from a computed imbalance of 0.9912476123528101 to a new imbalance of 4.2829815420000605E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6796296296296296, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-18 21:58:45,414 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1009127412:90 , srv1024701060:90 , srv1033478909:90 , srv1144184729:90 , srv1164087204:90 , srv1180414629:90 , srv1226355896:90 , srv1370481224:90 , srv1471198100:90 , srv1784555884:90 , srv1843890911:90 , srv1917423045:90 , srv1917449428:90 , srv1956288146:90 , srv1957196110:90 , srv1993831249:90 , srv2053379455:90 , srv213623998:90 , srv233243049:90 , srv423844926:90 , srv446805495:90 , srv573148990:90 , srv591512608:90 , srv603506708:90 , srv728970617:90 , srv786541438:90 , srv850174758:90 , srv877305018:90 , srv885192559:90 , srv949066951:90 }
2023-08-18 21:58:45,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-08-18 21:58:45,415 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv2053379455=16, srv233243049=18, srv1009127412=0, srv1784555884=9, srv591512608=22, srv573148990=21, srv1843890911=10, srv1024701060=1, srv885192559=28, srv1917449428=12, srv1164087204=4, srv1226355896=6, srv1956288146=13, srv728970617=24, srv1144184729=3, srv1033478909=2, srv1370481224=7, srv1957196110=14, srv423844926=19, srv1180414629=5, srv213623998=17, srv786541438=25, srv446805495=20, srv1471198100=8, srv1917423045=11, srv877305018=27, srv603506708=23, srv850174758=26, srv1993831249=15, srv949066951=29} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-08-18 21:58:45,419 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-08-18 21:58:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-08-18 21:58:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-08-18 21:58:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-08-18 21:58:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-08-18 21:58:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-08-18 21:58:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-08-18 21:58:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-08-18 21:58:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-08-18 21:58:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-08-18 21:58:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-08-18 21:58:45,420 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-08-18 21:58:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-08-18 21:58:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-08-18 21:58:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-08-18 21:58:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-08-18 21:58:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-08-18 21:58:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-08-18 21:58:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-08-18 21:58:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-08-18 21:58:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-08-18 21:58:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-08-18 21:58:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-08-18 21:58:45,421 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-08-18 21:58:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-08-18 21:58:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-08-18 21:58:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-08-18 21:58:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-08-18 21:58:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-08-18 21:58:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-08-18 21:58:45,422 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-08-18 21:58:45,436 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-08-18 21:58:45,437 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1009127412:90 , srv1024701060:90 , srv1033478909:90 , srv1144184729:90 , srv1164087204:90 , srv1180414629:90 , srv1226355896:90 , srv1370481224:90 , srv1471198100:90 , srv1784555884:90 , srv1843890911:90 , srv1917423045:90 , srv1917449428:90 , srv1956288146:90 , srv1957196110:90 , srv1993831249:90 , srv2053379455:90 , srv213623998:90 , srv233243049:90 , srv423844926:90 , srv446805495:90 , srv573148990:90 , srv591512608:90 , srv603506708:90 , srv728970617:90 , srv786541438:90 , srv850174758:90 , srv877305018:90 , srv885192559:90 , srv949066951:90 }
2023-08-18 21:58:45,449 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=181 (was 240), ProcessCount=166 (was 168), AvailableMemoryMB=2950 (was 3349)