2023-10-04 01:53:24,251 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-10-04 01:53:24,553 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-10-04 01:53:24,664 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-10-04 01:53:24,720 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-04 01:53:24,727 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-10-04 01:53:24,740 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=252, ProcessCount=167, AvailableMemoryMB=12350
2023-10-04 01:53:24,742 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-04 01:53:24,742 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-10-04 01:53:24,871 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1006386774:84 , srv1010396313:84 , srv101236160:84 , srv1024761107:84 , srv1070694927:84 , srv1085373749:84 , srv1112040270:84 , srv1141892366:84 , srv1170749926:84 , srv1177342158:84 , srv1283491169:84 , srv1289267505:84 , srv1291120934:84 , srv134077593:84 , srv1357855760:84 , srv1367887061:84 , srv1368824519:84 , srv139583538:84 , srv1418805945:84 , srv1475354580:84 , srv1543194817:84 , srv1548934238:84 , srv155886075:84 , srv1570968618:84 , srv1579305223:84 , srv159317893:84 , srv1603355489:84 , srv1623485260:84 , srv163370024:84 , srv1641394459:84 , srv1644643079:84 , srv1653732058:84 , srv1662145957:84 , srv1710530346:84 , srv1728838028:84 , srv1741092674:84 , srv1760231873:84 , srv178648331:84 , srv1806588554:84 , srv1854907473:84 , srv1863424560:84 , srv1877688890:84 , srv1883487542:84 , srv1889792328:84 , srv1895379257:84 , srv1908021780:84 , srv198623443:84 , srv2020650224:84 , srv2022741521:84 , srv2048351876:84 , srv2120326703:84 , srv2126619538:84 , srv2141761172:84 , srv222698083:84 , srv247005039:84 , srv316103864:84 , srv332846649:84 , srv347632482:84 , srv353063937:84 , srv360631824:84 , srv365227813:84 , srv367816946:84 , srv398803800:84 , srv405526001:84 , srv406197834:84 , srv426470272:84 , srv43377087:84 , srv444227348:84 , srv448378920:84 , srv491499361:84 , srv494854096:84 , srv517649512:84 , srv522095505:84 , srv523239673:84 , srv523578395:84 , srv584306691:84 , srv590512027:84 , srv597991388:84 , srv604366722:84 , srv617856150:84 , srv623127081:84 , srv625486579:84 , srv644390792:84 , srv653804498:84 , srv714618572:84 , srv729173664:84 , srv729381021:84 , srv751644363:84 , srv752778793:84 , srv776923228:84 , srv781433505:84 , srv803727208:84 , srv822258597:84 , srv83652718:84 , srv866201663:84 , srv8700134:84 , srv923510626:84 , srv946362342:84 , srv990914624:84 , srv739536336:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-10-04 01:53:24,879 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-04 01:53:24,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1854907473=39, srv2141761172=52, srv1908021780=45, srv1895379257=44, srv332846649=56, srv1883487542=42, srv426470272=65, srv751644363=88, srv1070694927=4, srv729381021=86, srv2048351876=49, srv781433505=91, srv353063937=58, srv491499361=69, srv1283491169=10, srv1710530346=33, srv803727208=92, srv316103864=55, srv2126619538=51, srv1728838028=34, srv405526001=63, srv247005039=54, srv444227348=67, srv739536336=87, srv347632482=57, srv2022741521=48, srv1579305223=24, srv822258597=93, srv139583538=17, srv1475354580=19, srv198623443=46, srv1418805945=18, srv1291120934=12, srv155886075=22, srv990914624=99, srv2120326703=50, srv522095505=72, srv1641394459=29, srv866201663=95, srv1006386774=0, srv83652718=94, srv1603355489=26, srv1112040270=6, srv653804498=83, srv1543194817=20, srv1177342158=9, srv1367887061=15, srv367816946=61, srv1357855760=14, srv1024761107=3, srv1141892366=7, srv776923228=90, srv1877688890=41, srv729173664=85, srv43377087=66, srv523578395=74, srv1863424560=40, srv159317893=25, srv1760231873=36, srv222698083=53, srv590512027=76, srv946362342=98, srv163370024=28, srv523239673=73, srv1368824519=16, srv1653732058=31, srv448378920=68, srv1806588554=38, srv1085373749=5, srv584306691=75, srv1548934238=21, srv597991388=77, srv604366722=78, srv406197834=64, srv923510626=97, srv1570968618=23, srv617856150=79, srv1889792328=43, srv101236160=2, srv494854096=70, srv1741092674=35, srv714618572=84, srv644390792=82, srv1170749926=8, srv1623485260=27, srv178648331=37, srv1644643079=30, srv517649512=71, srv8700134=96, srv365227813=60, srv1662145957=32, srv360631824=59, srv2020650224=47, srv623127081=80, srv752778793=89, srv398803800=62, srv1010396313=1, srv1289267505=11, srv625486579=81, srv134077593=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-04 01:53:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-04 01:53:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-04 01:53:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-04 01:53:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-04 01:53:24,957 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-04 01:53:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-04 01:53:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-04 01:53:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-04 01:53:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-04 01:53:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-04 01:53:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-04 01:53:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-04 01:53:24,958 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-04 01:53:24,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-04 01:53:24,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-04 01:53:24,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-04 01:53:24,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-04 01:53:24,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-04 01:53:24,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-04 01:53:24,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-04 01:53:24,959 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-04 01:53:24,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-04 01:53:24,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-04 01:53:24,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-04 01:53:24,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-04 01:53:24,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-04 01:53:24,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-04 01:53:24,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-04 01:53:24,960 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-04 01:53:24,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-04 01:53:24,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-04 01:53:24,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-04 01:53:24,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-04 01:53:24,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-04 01:53:24,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-04 01:53:24,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-04 01:53:24,961 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-04 01:53:24,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-04 01:53:24,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-04 01:53:24,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-04 01:53:24,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-04 01:53:24,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-04 01:53:24,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-04 01:53:24,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-04 01:53:24,962 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-04 01:53:24,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-04 01:53:24,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-04 01:53:24,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-04 01:53:24,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-04 01:53:24,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-04 01:53:24,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-04 01:53:24,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-04 01:53:24,963 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-04 01:53:24,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-04 01:53:24,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-04 01:53:24,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-04 01:53:24,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-04 01:53:24,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-04 01:53:24,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-04 01:53:24,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-04 01:53:24,964 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-04 01:53:24,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-04 01:53:24,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-04 01:53:24,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-04 01:53:24,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-04 01:53:24,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-04 01:53:24,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-04 01:53:24,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-04 01:53:24,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-04 01:53:24,965 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-04 01:53:24,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-04 01:53:24,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-04 01:53:24,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-04 01:53:24,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-04 01:53:24,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-04 01:53:24,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-04 01:53:24,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-04 01:53:24,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-04 01:53:24,966 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-04 01:53:24,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-04 01:53:24,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-04 01:53:24,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-04 01:53:24,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-04 01:53:24,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-04 01:53:24,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-04 01:53:24,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-04 01:53:24,967 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-04 01:53:24,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-04 01:53:24,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-04 01:53:24,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-04 01:53:24,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-04 01:53:24,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-04 01:53:24,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-04 01:53:24,968 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-04 01:53:24,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-04 01:53:24,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-04 01:53:24,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-04 01:53:24,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-04 01:53:24,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-04 01:53:24,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-04 01:53:24,969 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-04 01:53:25,321 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-04 01:53:25,321 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-04 01:53:25,322 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-04 01:53:26,135 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 1140 ms to try 5000 different iterations.  Found a solution that moves 4705 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1777134312172634. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03844787616946982, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05795814891638355, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5227777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03844787616946982, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23533333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-04 01:53:26,163 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1291120934:81 , srv1883487542:81 , srv752778793:81 , srv1006386774:82 , srv198623443:82 , srv43377087:82 , srv494854096:82 , srv523239673:82 , srv751644363:82 , srv776923228:82 , srv1070694927:83 , srv1662145957:83 , srv1806588554:83 , srv1877688890:83 , srv353063937:83 , srv523578395:83 , srv803727208:83 , srv1368824519:84 , srv155886075:84 , srv159317893:84 , srv1653732058:84 , srv1728838028:84 , srv2020650224:84 , srv398803800:84 , srv590512027:84 , srv617856150:84 , srv644390792:84 , srv653804498:84 , srv729381021:84 , srv866201663:84 , srv1085373749:85 , srv1141892366:85 , srv134077593:85 , srv139583538:85 , srv1548934238:85 , srv1644643079:85 , srv1908021780:85 , srv222698083:85 , srv332846649:85 , srv365227813:85 , srv426470272:85 , srv448378920:85 , srv923510626:85 , srv1418805945:86 , srv1579305223:86 , srv1710530346:86 , srv1760231873:86 , srv2022741521:86 , srv2048351876:86 , srv2141761172:86 , srv367816946:86 , srv405526001:86 , srv597991388:86 , srv729173664:86 , srv946362342:86 , srv101236160:87 , srv1112040270:87 , srv1367887061:87 , srv1603355489:87 , srv1854907473:87 , srv1863424560:87 , srv2120326703:87 , srv247005039:87 , srv347632482:87 , srv360631824:87 , srv406197834:87 , srv1283491169:88 , srv1641394459:88 , srv1895379257:88 , srv623127081:88 , srv714618572:88 , srv1010396313:89 , srv1357855760:89 , srv178648331:89 , srv491499361:89 , srv625486579:89 , srv1170749926:90 , srv1475354580:90 , srv1543194817:90 , srv1623485260:90 , srv1741092674:90 , srv316103864:90 , srv517649512:90 , srv522095505:90 , srv604366722:90 , srv990914624:90 , srv1177342158:91 , srv1289267505:91 , srv444227348:91 , srv822258597:91 , srv83652718:91 , srv8700134:91 , srv163370024:92 , srv2126619538:92 , srv781433505:92 , srv1889792328:93 , srv584306691:93 , srv1024761107:94 , srv1570968618:98 , srv739536336:431 }
2023-10-04 01:53:26,167 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-04 01:53:26,168 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1854907473=39, srv2141761172=52, srv1908021780=45, srv1895379257=44, srv332846649=56, srv1883487542=42, srv426470272=65, srv751644363=88, srv1070694927=4, srv729381021=86, srv2048351876=49, srv781433505=91, srv353063937=58, srv491499361=69, srv1283491169=10, srv1710530346=33, srv803727208=92, srv316103864=55, srv2126619538=51, srv1728838028=34, srv405526001=63, srv247005039=54, srv444227348=67, srv739536336=87, srv347632482=57, srv2022741521=48, srv1579305223=24, srv822258597=93, srv139583538=17, srv1475354580=19, srv198623443=46, srv1418805945=18, srv1291120934=12, srv155886075=22, srv990914624=99, srv2120326703=50, srv522095505=72, srv1641394459=29, srv866201663=95, srv1006386774=0, srv83652718=94, srv1603355489=26, srv1112040270=6, srv653804498=83, srv1543194817=20, srv1177342158=9, srv1367887061=15, srv367816946=61, srv1357855760=14, srv1024761107=3, srv1141892366=7, srv776923228=90, srv1877688890=41, srv729173664=85, srv43377087=66, srv523578395=74, srv1863424560=40, srv159317893=25, srv1760231873=36, srv222698083=53, srv590512027=76, srv946362342=98, srv163370024=28, srv523239673=73, srv1368824519=16, srv1653732058=31, srv448378920=68, srv1806588554=38, srv1085373749=5, srv584306691=75, srv1548934238=21, srv597991388=77, srv604366722=78, srv406197834=64, srv923510626=97, srv1570968618=23, srv617856150=79, srv1889792328=43, srv101236160=2, srv494854096=70, srv1741092674=35, srv714618572=84, srv644390792=82, srv1170749926=8, srv1623485260=27, srv178648331=37, srv1644643079=30, srv517649512=71, srv8700134=96, srv365227813=60, srv1662145957=32, srv360631824=59, srv2020650224=47, srv623127081=80, srv752778793=89, srv398803800=62, srv1010396313=1, srv1289267505=11, srv625486579=81, srv134077593=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-04 01:53:26,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-04 01:53:26,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-04 01:53:26,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-04 01:53:26,195 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-04 01:53:26,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-04 01:53:26,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-04 01:53:26,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-04 01:53:26,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-04 01:53:26,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-04 01:53:26,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-04 01:53:26,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-04 01:53:26,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-04 01:53:26,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-04 01:53:26,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-04 01:53:26,196 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-04 01:53:26,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-04 01:53:26,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-04 01:53:26,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-04 01:53:26,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-04 01:53:26,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-04 01:53:26,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-04 01:53:26,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-04 01:53:26,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-04 01:53:26,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-04 01:53:26,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-04 01:53:26,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-04 01:53:26,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-04 01:53:26,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-04 01:53:26,197 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-04 01:53:26,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-04 01:53:26,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-04 01:53:26,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-04 01:53:26,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-04 01:53:26,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-04 01:53:26,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-04 01:53:26,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-04 01:53:26,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-04 01:53:26,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-04 01:53:26,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-04 01:53:26,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-04 01:53:26,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-04 01:53:26,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-04 01:53:26,198 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-04 01:53:26,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-04 01:53:26,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-04 01:53:26,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-04 01:53:26,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-04 01:53:26,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-04 01:53:26,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-04 01:53:26,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-04 01:53:26,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-04 01:53:26,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-04 01:53:26,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-04 01:53:26,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-04 01:53:26,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-04 01:53:26,199 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-04 01:53:26,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-04 01:53:26,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-04 01:53:26,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-04 01:53:26,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-04 01:53:26,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-04 01:53:26,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-04 01:53:26,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-04 01:53:26,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-04 01:53:26,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-04 01:53:26,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-04 01:53:26,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-04 01:53:26,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-04 01:53:26,200 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-04 01:53:26,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-04 01:53:26,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-04 01:53:26,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-04 01:53:26,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-04 01:53:26,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-04 01:53:26,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-04 01:53:26,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-04 01:53:26,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-04 01:53:26,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-04 01:53:26,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-04 01:53:26,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-04 01:53:26,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-04 01:53:26,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-04 01:53:26,201 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-04 01:53:26,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-04 01:53:26,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-04 01:53:26,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-04 01:53:26,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-04 01:53:26,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-04 01:53:26,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-04 01:53:26,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-04 01:53:26,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-04 01:53:26,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-04 01:53:26,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-04 01:53:26,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-04 01:53:26,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-04 01:53:26,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-04 01:53:26,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-04 01:53:26,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-04 01:53:26,202 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-04 01:53:26,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-04 01:53:26,203 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-04 01:53:26,386 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03844787616946982, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05795814891638355, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03844787616946982, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23533333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-04 01:53:26,387 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-04 01:53:26,387 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17768048610311338, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03844787616946982, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05795814891638355, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.03844787616946982, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17333333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.23533333333333334, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-04 01:53:26,743 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 522 ms to try 5000 different iterations.  Found a solution that moves 2585 regions; Going from a computed imbalance of 0.17768048610311338 to a new imbalance of 0.014389139346584536. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02028233194558177); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037548322774924, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.2872222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02028233194558177); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-04 01:53:26,755 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1570968618:74 , srv1024761107:77 , srv1889792328:78 , srv584306691:78 , srv1177342158:79 , srv2126619538:79 , srv1289267505:80 , srv1543194817:80 , srv1623485260:80 , srv163370024:80 , srv781433505:80 , srv990914624:80 , srv1475354580:81 , srv491499361:81 , srv83652718:81 , srv8700134:81 , srv1170749926:82 , srv1741092674:82 , srv316103864:82 , srv1579305223:83 , srv1641394459:83 , srv2120326703:83 , srv444227348:83 , srv517649512:83 , srv714618572:83 , srv1010396313:84 , srv1283491169:84 , srv1367887061:84 , srv1603355489:84 , srv1895379257:84 , srv406197834:84 , srv522095505:84 , srv604366722:84 , srv623127081:84 , srv101236160:85 , srv1357855760:85 , srv178648331:85 , srv1854907473:85 , srv1863424560:85 , srv2022741521:85 , srv2141761172:85 , srv625486579:85 , srv729173664:85 , srv159317893:86 , srv1877688890:86 , srv222698083:86 , srv247005039:86 , srv653804498:86 , srv1070694927:87 , srv1085373749:87 , srv1112040270:87 , srv139583538:87 , srv1418805945:87 , srv1548934238:87 , srv1760231873:87 , srv1908021780:87 , srv2020650224:87 , srv590512027:87 , srv597991388:87 , srv644390792:87 , srv803727208:87 , srv946362342:87 , srv1141892366:88 , srv134077593:88 , srv1710530346:88 , srv1728838028:88 , srv332846649:88 , srv347632482:88 , srv360631824:88 , srv365227813:88 , srv367816946:88 , srv398803800:88 , srv405526001:88 , srv426470272:88 , srv448378920:88 , srv523239673:88 , srv617856150:88 , srv751644363:88 , srv866201663:88 , srv1368824519:89 , srv1653732058:89 , srv1662145957:89 , srv1806588554:89 , srv2048351876:89 , srv822258597:89 , srv923510626:89 , srv1006386774:90 , srv155886075:90 , srv1644643079:90 , srv1883487542:90 , srv198623443:90 , srv353063937:90 , srv494854096:90 , srv523578395:90 , srv729381021:90 , srv1291120934:91 , srv43377087:91 , srv752778793:91 , srv776923228:93 , srv739536336:523 }
2023-10-04 01:53:26,758 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-04 01:53:26,759 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1854907473=39, srv2141761172=52, srv1908021780=45, srv1895379257=44, srv332846649=56, srv1883487542=42, srv426470272=65, srv751644363=88, srv1070694927=4, srv729381021=86, srv2048351876=49, srv781433505=91, srv353063937=58, srv491499361=69, srv1283491169=10, srv1710530346=33, srv803727208=92, srv316103864=55, srv2126619538=51, srv1728838028=34, srv405526001=63, srv247005039=54, srv444227348=67, srv739536336=87, srv347632482=57, srv2022741521=48, srv1579305223=24, srv822258597=93, srv139583538=17, srv1475354580=19, srv198623443=46, srv1418805945=18, srv1291120934=12, srv155886075=22, srv990914624=99, srv2120326703=50, srv522095505=72, srv1641394459=29, srv866201663=95, srv1006386774=0, srv83652718=94, srv1603355489=26, srv1112040270=6, srv653804498=83, srv1543194817=20, srv1177342158=9, srv1367887061=15, srv367816946=61, srv1357855760=14, srv1024761107=3, srv1141892366=7, srv776923228=90, srv1877688890=41, srv729173664=85, srv43377087=66, srv523578395=74, srv1863424560=40, srv159317893=25, srv1760231873=36, srv222698083=53, srv590512027=76, srv946362342=98, srv163370024=28, srv523239673=73, srv1368824519=16, srv1653732058=31, srv448378920=68, srv1806588554=38, srv1085373749=5, srv584306691=75, srv1548934238=21, srv597991388=77, srv604366722=78, srv406197834=64, srv923510626=97, srv1570968618=23, srv617856150=79, srv1889792328=43, srv101236160=2, srv494854096=70, srv1741092674=35, srv714618572=84, srv644390792=82, srv1170749926=8, srv1623485260=27, srv178648331=37, srv1644643079=30, srv517649512=71, srv8700134=96, srv365227813=60, srv1662145957=32, srv360631824=59, srv2020650224=47, srv623127081=80, srv752778793=89, srv398803800=62, srv1010396313=1, srv1289267505=11, srv625486579=81, srv134077593=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-04 01:53:26,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-04 01:53:26,785 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-04 01:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-04 01:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-04 01:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-04 01:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-04 01:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-04 01:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-04 01:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-04 01:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-04 01:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-04 01:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-04 01:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-04 01:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-04 01:53:26,786 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-04 01:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-04 01:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-04 01:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-04 01:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-04 01:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-04 01:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-04 01:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-04 01:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-04 01:53:26,787 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-04 01:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-04 01:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-04 01:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-04 01:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-04 01:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-04 01:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-04 01:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-04 01:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-04 01:53:26,788 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-04 01:53:26,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-04 01:53:26,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-04 01:53:26,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-04 01:53:26,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-04 01:53:26,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-04 01:53:26,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-04 01:53:26,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-04 01:53:26,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-04 01:53:26,789 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-04 01:53:26,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-04 01:53:26,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-04 01:53:26,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-04 01:53:26,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-04 01:53:26,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-04 01:53:26,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-04 01:53:26,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-04 01:53:26,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-04 01:53:26,790 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-04 01:53:26,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-04 01:53:26,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-04 01:53:26,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-04 01:53:26,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-04 01:53:26,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-04 01:53:26,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-04 01:53:26,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-04 01:53:26,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-04 01:53:26,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-04 01:53:26,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-04 01:53:26,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-04 01:53:26,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-04 01:53:26,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-04 01:53:26,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-04 01:53:26,791 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-04 01:53:26,792 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-04 01:53:26,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-04 01:53:26,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-04 01:53:26,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-04 01:53:26,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-04 01:53:26,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-04 01:53:26,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-04 01:53:26,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-04 01:53:26,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-04 01:53:26,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-04 01:53:26,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-04 01:53:26,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-04 01:53:26,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-04 01:53:26,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-04 01:53:26,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-04 01:53:26,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-04 01:53:26,793 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-04 01:53:26,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-04 01:53:26,794 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-04 01:53:26,908 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02028233194558177); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037548322774924, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02028233194558177); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-04 01:53:26,909 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-04 01:53:26,909 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.014371038789713577, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02028233194558177); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.037548322774924, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02028233194558177); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011333333333333334); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.043333333333333335, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-04 01:53:27,261 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 454 ms to try 5000 different iterations.  Found a solution that moves 1001 regions; Going from a computed imbalance of 0.014371038789713577 to a new imbalance of 6.118758466614781E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0047588039714625595); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010948302145324627); MoveCostFunction : (multiplier=7.0, imbalance=0.11122222222222222, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0047588039714625595); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-04 01:53:27,264 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv822258597:78 , srv347632482:82 , srv360631824:82 , srv1357855760:83 , srv1644643079:83 , srv1710530346:83 , srv2048351876:83 , srv405526001:83 , srv444227348:83 , srv625486579:83 , srv776923228:83 , srv923510626:83 , srv1010396313:84 , srv1112040270:84 , srv1141892366:84 , srv134077593:84 , srv155886075:84 , srv1653732058:84 , srv1760231873:84 , srv178648331:84 , srv247005039:84 , srv332846649:84 , srv367816946:84 , srv448378920:84 , srv522095505:84 , srv604366722:84 , srv729381021:84 , srv1368824519:85 , srv139583538:85 , srv1418805945:85 , srv1548934238:85 , srv1662145957:85 , srv1741092674:85 , srv1806588554:85 , srv1895379257:85 , srv353063937:85 , srv365227813:85 , srv398803800:85 , srv426470272:85 , srv43377087:85 , srv494854096:85 , srv517649512:85 , srv523578395:85 , srv597991388:85 , srv752778793:85 , srv781433505:85 , srv866201663:85 , srv946362342:85 , srv1006386774:86 , srv101236160:86 , srv1024761107:86 , srv1085373749:86 , srv1170749926:86 , srv1283491169:86 , srv1289267505:86 , srv1291120934:86 , srv1367887061:86 , srv1475354580:86 , srv1570968618:86 , srv1728838028:86 , srv1854907473:86 , srv1863424560:86 , srv1908021780:86 , srv198623443:86 , srv2020650224:86 , srv2022741521:86 , srv2126619538:86 , srv222698083:86 , srv316103864:86 , srv523239673:86 , srv584306691:86 , srv617856150:86 , srv623127081:86 , srv729173664:86 , srv83652718:86 , srv8700134:86 , srv1543194817:87 , srv1603355489:87 , srv1623485260:87 , srv163370024:87 , srv2120326703:87 , srv2141761172:87 , srv406197834:87 , srv590512027:87 , srv644390792:87 , srv653804498:87 , srv714618572:87 , srv751644363:87 , srv990914624:87 , srv1070694927:88 , srv1177342158:88 , srv1579305223:88 , srv159317893:88 , srv1641394459:88 , srv1883487542:88 , srv1889792328:88 , srv491499361:88 , srv803727208:88 , srv1877688890:89 , srv739536336:546 }
2023-10-04 01:53:27,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-04 01:53:27,271 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1854907473=39, srv2141761172=52, srv1908021780=45, srv1895379257=44, srv332846649=56, srv1883487542=42, srv426470272=65, srv751644363=88, srv1070694927=4, srv729381021=86, srv2048351876=49, srv781433505=91, srv353063937=58, srv491499361=69, srv1283491169=10, srv1710530346=33, srv803727208=92, srv316103864=55, srv2126619538=51, srv1728838028=34, srv405526001=63, srv247005039=54, srv444227348=67, srv739536336=87, srv347632482=57, srv2022741521=48, srv1579305223=24, srv822258597=93, srv139583538=17, srv1475354580=19, srv198623443=46, srv1418805945=18, srv1291120934=12, srv155886075=22, srv990914624=99, srv2120326703=50, srv522095505=72, srv1641394459=29, srv866201663=95, srv1006386774=0, srv83652718=94, srv1603355489=26, srv1112040270=6, srv653804498=83, srv1543194817=20, srv1177342158=9, srv1367887061=15, srv367816946=61, srv1357855760=14, srv1024761107=3, srv1141892366=7, srv776923228=90, srv1877688890=41, srv729173664=85, srv43377087=66, srv523578395=74, srv1863424560=40, srv159317893=25, srv1760231873=36, srv222698083=53, srv590512027=76, srv946362342=98, srv163370024=28, srv523239673=73, srv1368824519=16, srv1653732058=31, srv448378920=68, srv1806588554=38, srv1085373749=5, srv584306691=75, srv1548934238=21, srv597991388=77, srv604366722=78, srv406197834=64, srv923510626=97, srv1570968618=23, srv617856150=79, srv1889792328=43, srv101236160=2, srv494854096=70, srv1741092674=35, srv714618572=84, srv644390792=82, srv1170749926=8, srv1623485260=27, srv178648331=37, srv1644643079=30, srv517649512=71, srv8700134=96, srv365227813=60, srv1662145957=32, srv360631824=59, srv2020650224=47, srv623127081=80, srv752778793=89, srv398803800=62, srv1010396313=1, srv1289267505=11, srv625486579=81, srv134077593=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-04 01:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-04 01:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-04 01:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-04 01:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-04 01:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-04 01:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-04 01:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-04 01:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-04 01:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-04 01:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-04 01:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-04 01:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-04 01:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-04 01:53:27,290 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-04 01:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-04 01:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-04 01:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-04 01:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-04 01:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-04 01:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-04 01:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-04 01:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-04 01:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-04 01:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-04 01:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-04 01:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-04 01:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-04 01:53:27,291 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-04 01:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-04 01:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-04 01:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-04 01:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-04 01:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-04 01:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-04 01:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-04 01:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-04 01:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-04 01:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-04 01:53:27,292 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-04 01:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-04 01:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-04 01:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-04 01:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-04 01:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-04 01:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-04 01:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-04 01:53:27,293 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-04 01:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-04 01:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-04 01:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-04 01:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-04 01:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-04 01:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-04 01:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-04 01:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-04 01:53:27,294 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-04 01:53:27,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-04 01:53:27,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-04 01:53:27,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-04 01:53:27,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-04 01:53:27,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-04 01:53:27,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-04 01:53:27,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-04 01:53:27,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-04 01:53:27,295 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-04 01:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-04 01:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-04 01:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-04 01:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-04 01:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-04 01:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-04 01:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-04 01:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-04 01:53:27,296 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-04 01:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-04 01:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-04 01:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-04 01:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-04 01:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-04 01:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-04 01:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-04 01:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-04 01:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-04 01:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-04 01:53:27,297 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-04 01:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-04 01:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-04 01:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-04 01:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-04 01:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-04 01:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-04 01:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-04 01:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-04 01:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-04 01:53:27,298 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-04 01:53:27,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-04 01:53:27,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-04 01:53:27,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-04 01:53:27,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-04 01:53:27,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-04 01:53:27,299 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-04 01:53:27,436 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0047588039714625595); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010948302145324627); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0047588039714625595); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-04 01:53:27,436 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-10-04 01:53:27,436 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.048666948518726E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0047588039714625595); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010948302145324627); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0047588039714625595); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.003416666666666667); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-10-04 01:53:27,820 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 506 ms to try 5000 different iterations.  Found a solution that moves 238 regions; Going from a computed imbalance of 6.048666948518726E-4 to a new imbalance of 1.6948049987116227E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00276258471336916); MoveCostFunction : (multiplier=7.0, imbalance=0.026444444444444444, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-04 01:53:27,821 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1570968618:83 , srv163370024:83 , srv1641394459:83 , srv1854907473:83 , srv1889792328:83 , srv517649512:83 , srv617856150:83 , srv1006386774:84 , srv101236160:84 , srv1070694927:84 , srv1085373749:84 , srv1112040270:84 , srv1170749926:84 , srv1177342158:84 , srv1367887061:84 , srv1368824519:84 , srv1418805945:84 , srv159317893:84 , srv1603355489:84 , srv1644643079:84 , srv1653732058:84 , srv1728838028:84 , srv178648331:84 , srv1863424560:84 , srv1877688890:84 , srv1883487542:84 , srv1908021780:84 , srv198623443:84 , srv2126619538:84 , srv2141761172:84 , srv222698083:84 , srv316103864:84 , srv353063937:84 , srv360631824:84 , srv365227813:84 , srv367816946:84 , srv405526001:84 , srv426470272:84 , srv43377087:84 , srv444227348:84 , srv494854096:84 , srv522095505:84 , srv523578395:84 , srv590512027:84 , srv597991388:84 , srv604366722:84 , srv623127081:84 , srv644390792:84 , srv653804498:84 , srv714618572:84 , srv729381021:84 , srv803727208:84 , srv822258597:84 , srv83652718:84 , srv946362342:84 , srv1010396313:85 , srv1024761107:85 , srv1141892366:85 , srv1289267505:85 , srv1291120934:85 , srv134077593:85 , srv1357855760:85 , srv139583538:85 , srv1475354580:85 , srv1543194817:85 , srv1548934238:85 , srv155886075:85 , srv1579305223:85 , srv1623485260:85 , srv1662145957:85 , srv1710530346:85 , srv1741092674:85 , srv1760231873:85 , srv1895379257:85 , srv2020650224:85 , srv2022741521:85 , srv2048351876:85 , srv2120326703:85 , srv247005039:85 , srv332846649:85 , srv347632482:85 , srv406197834:85 , srv448378920:85 , srv491499361:85 , srv584306691:85 , srv729173664:85 , srv751644363:85 , srv752778793:85 , srv776923228:85 , srv781433505:85 , srv866201663:85 , srv8700134:85 , srv923510626:85 , srv990914624:85 , srv1283491169:86 , srv1806588554:86 , srv398803800:86 , srv523239673:86 , srv625486579:86 , srv739536336:642 }
2023-10-04 01:53:27,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-04 01:53:27,827 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1854907473=39, srv2141761172=52, srv1908021780=45, srv1895379257=44, srv332846649=56, srv1883487542=42, srv426470272=65, srv751644363=88, srv1070694927=4, srv729381021=86, srv2048351876=49, srv781433505=91, srv353063937=58, srv491499361=69, srv1283491169=10, srv1710530346=33, srv803727208=92, srv316103864=55, srv2126619538=51, srv1728838028=34, srv405526001=63, srv247005039=54, srv444227348=67, srv739536336=87, srv347632482=57, srv2022741521=48, srv1579305223=24, srv822258597=93, srv139583538=17, srv1475354580=19, srv198623443=46, srv1418805945=18, srv1291120934=12, srv155886075=22, srv990914624=99, srv2120326703=50, srv522095505=72, srv1641394459=29, srv866201663=95, srv1006386774=0, srv83652718=94, srv1603355489=26, srv1112040270=6, srv653804498=83, srv1543194817=20, srv1177342158=9, srv1367887061=15, srv367816946=61, srv1357855760=14, srv1024761107=3, srv1141892366=7, srv776923228=90, srv1877688890=41, srv729173664=85, srv43377087=66, srv523578395=74, srv1863424560=40, srv159317893=25, srv1760231873=36, srv222698083=53, srv590512027=76, srv946362342=98, srv163370024=28, srv523239673=73, srv1368824519=16, srv1653732058=31, srv448378920=68, srv1806588554=38, srv1085373749=5, srv584306691=75, srv1548934238=21, srv597991388=77, srv604366722=78, srv406197834=64, srv923510626=97, srv1570968618=23, srv617856150=79, srv1889792328=43, srv101236160=2, srv494854096=70, srv1741092674=35, srv714618572=84, srv644390792=82, srv1170749926=8, srv1623485260=27, srv178648331=37, srv1644643079=30, srv517649512=71, srv8700134=96, srv365227813=60, srv1662145957=32, srv360631824=59, srv2020650224=47, srv623127081=80, srv752778793=89, srv398803800=62, srv1010396313=1, srv1289267505=11, srv625486579=81, srv134077593=13} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-04 01:53:27,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-04 01:53:27,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-04 01:53:27,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-04 01:53:27,845 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-04 01:53:27,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-04 01:53:27,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-04 01:53:27,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-04 01:53:27,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-04 01:53:27,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-04 01:53:27,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-04 01:53:27,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-04 01:53:27,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-04 01:53:27,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-04 01:53:27,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-04 01:53:27,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-04 01:53:27,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-04 01:53:27,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-04 01:53:27,846 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-04 01:53:27,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-04 01:53:27,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-04 01:53:27,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-04 01:53:27,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-04 01:53:27,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-04 01:53:27,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-04 01:53:27,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-04 01:53:27,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-04 01:53:27,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-04 01:53:27,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-04 01:53:27,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-04 01:53:27,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-04 01:53:27,847 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-10-04 01:53:27,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-10-04 01:53:27,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-10-04 01:53:27,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-10-04 01:53:27,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-10-04 01:53:27,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-10-04 01:53:27,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-10-04 01:53:27,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-10-04 01:53:27,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-10-04 01:53:27,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-10-04 01:53:27,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-10-04 01:53:27,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-10-04 01:53:27,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-10-04 01:53:27,848 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-10-04 01:53:27,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-10-04 01:53:27,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-10-04 01:53:27,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-10-04 01:53:27,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-10-04 01:53:27,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-10-04 01:53:27,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-10-04 01:53:27,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-10-04 01:53:27,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-10-04 01:53:27,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-10-04 01:53:27,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-10-04 01:53:27,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-10-04 01:53:27,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-10-04 01:53:27,849 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-10-04 01:53:27,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-10-04 01:53:27,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-10-04 01:53:27,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-10-04 01:53:27,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-10-04 01:53:27,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-10-04 01:53:27,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-10-04 01:53:27,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-10-04 01:53:27,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-10-04 01:53:27,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-10-04 01:53:27,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-10-04 01:53:27,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-10-04 01:53:27,850 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-10-04 01:53:27,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-10-04 01:53:27,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-10-04 01:53:27,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-10-04 01:53:27,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-10-04 01:53:27,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-10-04 01:53:27,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-10-04 01:53:27,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-10-04 01:53:27,851 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-10-04 01:53:27,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-10-04 01:53:27,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-10-04 01:53:27,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-10-04 01:53:27,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-10-04 01:53:27,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-10-04 01:53:27,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-10-04 01:53:27,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-10-04 01:53:27,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-10-04 01:53:27,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-10-04 01:53:27,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-10-04 01:53:27,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-10-04 01:53:27,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-10-04 01:53:27,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-10-04 01:53:27,852 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-10-04 01:53:27,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-10-04 01:53:27,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-10-04 01:53:27,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-10-04 01:53:27,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-10-04 01:53:27,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-10-04 01:53:27,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-10-04 01:53:27,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-10-04 01:53:27,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-10-04 01:53:27,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-10-04 01:53:27,853 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-10-04 01:53:28,003 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.528153836804917E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.909066927536782E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00276258471336916); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.909066927536782E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-04 01:53:28,003 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1570968618:83 , srv163370024:83 , srv1641394459:83 , srv1854907473:83 , srv1889792328:83 , srv517649512:83 , srv617856150:83 , srv1006386774:84 , srv101236160:84 , srv1070694927:84 , srv1085373749:84 , srv1112040270:84 , srv1170749926:84 , srv1177342158:84 , srv1367887061:84 , srv1368824519:84 , srv1418805945:84 , srv159317893:84 , srv1603355489:84 , srv1644643079:84 , srv1653732058:84 , srv1728838028:84 , srv178648331:84 , srv1863424560:84 , srv1877688890:84 , srv1883487542:84 , srv1908021780:84 , srv198623443:84 , srv2126619538:84 , srv2141761172:84 , srv222698083:84 , srv316103864:84 , srv353063937:84 , srv360631824:84 , srv365227813:84 , srv367816946:84 , srv405526001:84 , srv426470272:84 , srv43377087:84 , srv444227348:84 , srv494854096:84 , srv522095505:84 , srv523578395:84 , srv590512027:84 , srv597991388:84 , srv604366722:84 , srv623127081:84 , srv644390792:84 , srv653804498:84 , srv714618572:84 , srv729381021:84 , srv803727208:84 , srv822258597:84 , srv83652718:84 , srv946362342:84 , srv1010396313:85 , srv1024761107:85 , srv1141892366:85 , srv1289267505:85 , srv1291120934:85 , srv134077593:85 , srv1357855760:85 , srv139583538:85 , srv1475354580:85 , srv1543194817:85 , srv1548934238:85 , srv155886075:85 , srv1579305223:85 , srv1623485260:85 , srv1662145957:85 , srv1710530346:85 , srv1741092674:85 , srv1760231873:85 , srv1895379257:85 , srv2020650224:85 , srv2022741521:85 , srv2048351876:85 , srv2120326703:85 , srv247005039:85 , srv332846649:85 , srv347632482:85 , srv406197834:85 , srv448378920:85 , srv491499361:85 , srv584306691:85 , srv729173664:85 , srv751644363:85 , srv752778793:85 , srv776923228:85 , srv781433505:85 , srv866201663:85 , srv8700134:85 , srv923510626:85 , srv990914624:85 , srv1283491169:86 , srv1806588554:86 , srv398803800:86 , srv523239673:86 , srv625486579:86 , srv739536336:642 }
2023-10-04 01:53:28,034 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=296 (was 252) - SystemLoadAverage LEAK? -, ProcessCount=167 (was 167), AvailableMemoryMB=12340 (was 12350)
2023-10-04 01:53:28,042 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=263, MaxFileDescriptor=60000, SystemLoadAverage=296, ProcessCount=167, AvailableMemoryMB=12340
2023-10-04 01:53:28,043 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-10-04 01:53:28,044 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-10-04 01:53:28,060 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv101179543:84 , srv1166349331:84 , srv1248058254:84 , srv1281601745:84 , srv1332139668:84 , srv1372567655:84 , srv1448746743:84 , srv1466010225:84 , srv1580039038:84 , srv1582309217:84 , srv1616478653:84 , srv1620724807:84 , srv1633995278:84 , srv1664913411:84 , srv1740225909:84 , srv1740743699:84 , srv1893316153:84 , srv2062577234:84 , srv2064329734:84 , srv212773185:84 , srv586612740:84 , srv647524863:84 , srv662388407:84 , srv781017653:84 , srv806383356:84 , srv881157802:84 , srv924812590:84 , srv926297785:84 , srv938536801:84 , srv201254766:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-10-04 01:53:28,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-04 01:53:28,062 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1633995278=12, srv1664913411=13, srv1616478653=10, srv1580039038=8, srv881157802=26, srv212773185=20, srv1893316153=16, srv1332139668=4, srv2062577234=18, srv926297785=28, srv201254766=17, srv662388407=23, srv1740743699=15, srv924812590=27, srv2064329734=19, srv938536801=29, srv1166349331=1, srv1582309217=9, srv1248058254=2, srv647524863=22, srv781017653=24, srv1281601745=3, srv586612740=21, srv1466010225=7, srv1620724807=11, srv806383356=25, srv1372567655=5, srv1448746743=6, srv101179543=0, srv1740225909=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-04 01:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-04 01:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-04 01:53:28,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-04 01:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-04 01:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-04 01:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-04 01:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-04 01:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-04 01:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-04 01:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-04 01:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-04 01:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-04 01:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-04 01:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-04 01:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-04 01:53:28,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-04 01:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-04 01:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-04 01:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-04 01:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-04 01:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-04 01:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-04 01:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-04 01:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-04 01:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-04 01:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-04 01:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-04 01:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-04 01:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-04 01:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-04 01:53:28,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-10-04 01:53:28,102 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.081049690873159, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-04 01:53:28,102 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912448428193705, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.081049690873159, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-10-04 01:55:28,081 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6402969 different iterations.  Found a solution that moves 1858 regions; Going from a computed imbalance of 0.9912448428193705 to a new imbalance of 4.336664689392977E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6881481481481482, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-04 01:55:28,087 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv101179543:90 , srv1166349331:90 , srv1248058254:90 , srv1281601745:90 , srv1332139668:90 , srv1372567655:90 , srv1448746743:90 , srv1466010225:90 , srv1580039038:90 , srv1582309217:90 , srv1616478653:90 , srv1620724807:90 , srv1633995278:90 , srv1664913411:90 , srv1740225909:90 , srv1740743699:90 , srv1893316153:90 , srv201254766:90 , srv2062577234:90 , srv2064329734:90 , srv212773185:90 , srv586612740:90 , srv647524863:90 , srv662388407:90 , srv781017653:90 , srv806383356:90 , srv881157802:90 , srv924812590:90 , srv926297785:90 , srv938536801:90 }
2023-10-04 01:55:28,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-10-04 01:55:28,088 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1633995278=12, srv1664913411=13, srv1616478653=10, srv1580039038=8, srv881157802=26, srv212773185=20, srv1893316153=16, srv1332139668=4, srv2062577234=18, srv926297785=28, srv201254766=17, srv662388407=23, srv1740743699=15, srv924812590=27, srv2064329734=19, srv938536801=29, srv1166349331=1, srv1582309217=9, srv1248058254=2, srv647524863=22, srv781017653=24, srv1281601745=3, srv586612740=21, srv1466010225=7, srv1620724807=11, srv806383356=25, srv1372567655=5, srv1448746743=6, srv101179543=0, srv1740225909=14} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-10-04 01:55:28,091 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-10-04 01:55:28,092 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-10-04 01:55:28,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-10-04 01:55:28,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-10-04 01:55:28,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-10-04 01:55:28,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-10-04 01:55:28,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-10-04 01:55:28,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-10-04 01:55:28,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-10-04 01:55:28,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-10-04 01:55:28,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-10-04 01:55:28,093 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-10-04 01:55:28,104 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-10-04 01:55:28,105 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv101179543:90 , srv1166349331:90 , srv1248058254:90 , srv1281601745:90 , srv1332139668:90 , srv1372567655:90 , srv1448746743:90 , srv1466010225:90 , srv1580039038:90 , srv1582309217:90 , srv1616478653:90 , srv1620724807:90 , srv1633995278:90 , srv1664913411:90 , srv1740225909:90 , srv1740743699:90 , srv1893316153:90 , srv201254766:90 , srv2062577234:90 , srv2064329734:90 , srv212773185:90 , srv586612740:90 , srv647524863:90 , srv662388407:90 , srv781017653:90 , srv806383356:90 , srv881157802:90 , srv924812590:90 , srv926297785:90 , srv938536801:90 }
2023-10-04 01:55:28,117 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=263 (was 263), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=167 (was 296), ProcessCount=167 (was 167), AvailableMemoryMB=11120 (was 12340)