2024-01-04 09:53:16,284 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-01-04 09:53:16,617 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-01-04 09:53:16,737 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-01-04 09:53:16,805 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-04 09:53:16,813 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-04 09:53:16,828 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=304, ProcessCount=168, AvailableMemoryMB=11376
2024-01-04 09:53:16,830 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-04 09:53:16,831 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-01-04 09:53:16,969 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1051640485:84 , srv1064489619:84 , srv1088476612:84 , srv1110505602:84 , srv1124331355:84 , srv1133088850:84 , srv1140987548:84 , srv1158880794:84 , srv1168034411:84 , srv1212453446:84 , srv1220279209:84 , srv1229146344:84 , srv1234413888:84 , srv1237409763:84 , srv1253650368:84 , srv1259077383:84 , srv1309440237:84 , srv1321336267:84 , srv1328172477:84 , srv1333672068:84 , srv134874889:84 , srv1369851591:84 , srv1392214657:84 , srv1421344993:84 , srv1431431150:84 , srv1468772263:84 , srv1472492355:84 , srv1516624642:84 , srv1540446986:84 , srv1552114119:84 , srv1609009682:84 , srv1682358881:84 , srv1705873468:84 , srv1723066229:84 , srv1729291934:84 , srv174626398:84 , srv1753083489:84 , srv1760425567:84 , srv1767910349:84 , srv1783674610:84 , srv1827083822:84 , srv1854986208:84 , srv1857800905:84 , srv1869041242:84 , srv1886799389:84 , srv1889265964:84 , srv1891675792:84 , srv1892557721:84 , srv1953165476:84 , srv204849378:84 , srv2051205482:84 , srv2080382958:84 , srv2087960017:84 , srv2119590923:84 , srv2122106230:84 , srv2125726678:84 , srv228553675:84 , srv234040835:84 , srv235621702:84 , srv279375121:84 , srv289644545:84 , srv294029025:84 , srv298796407:84 , srv325384829:84 , srv328719532:84 , srv329338269:84 , srv329508097:84 , srv344883115:84 , srv361319893:84 , srv382148028:84 , srv397640445:84 , srv441935816:84 , srv463020622:84 , srv47968199:84 , srv521494836:84 , srv551143944:84 , srv561486382:84 , srv572976011:84 , srv614604855:84 , srv637628560:84 , srv665853997:84 , srv705684186:84 , srv712158209:84 , srv73145001:84 , srv741240449:84 , srv780460827:84 , srv787812609:84 , srv799026841:84 , srv842679828:84 , srv852345879:84 , srv861470529:84 , srv862419198:84 , srv877268363:84 , srv880710558:84 , srv88463185:84 , srv930074797:84 , srv951082267:84 , srv967968061:84 , srv981331916:84 , srv648937576:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-01-04 09:53:16,983 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-04 09:53:16,992 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1133088850=5, srv235621702=58, srv1886799389=44, srv705684186=82, srv1237409763=13, srv1723066229=33, srv1891675792=46, srv1760425567=37, srv1753083489=36, srv1431431150=24, srv1682358881=31, srv294029025=61, srv1889265964=45, srv328719532=64, srv1767910349=38, srv279375121=59, srv1854986208=41, srv1552114119=29, srv551143944=75, srv397640445=70, srv174626398=35, srv88463185=95, srv1309440237=16, srv325384829=63, srv47968199=73, srv1321336267=17, srv1953165476=48, srv1472492355=26, srv1869041242=43, srv951082267=97, srv1110505602=3, srv861470529=91, srv2122106230=54, srv614604855=78, srv298796407=62, srv2080382958=51, srv1468772263=25, srv572976011=77, srv134874889=20, srv2125726678=55, srv799026841=88, srv1729291934=34, srv1158880794=7, srv1392214657=22, srv1229146344=11, srv463020622=72, srv2051205482=50, srv862419198=92, srv1064489619=1, srv1333672068=19, srv2087960017=52, srv1540446986=28, srv234040835=57, srv842679828=89, srv967968061=98, srv1609009682=30, srv877268363=93, srv1168034411=8, srv780460827=86, srv1124331355=4, srv1857800905=42, srv1892557721=47, srv329338269=65, srv2119590923=53, srv382148028=69, srv73145001=84, srv1827083822=40, srv289644545=60, srv441935816=71, srv521494836=74, srv228553675=56, srv852345879=90, srv1140987548=6, srv1705873468=32, srv561486382=76, srv665853997=81, srv1516624642=27, srv1051640485=0, srv344883115=67, srv1328172477=18, srv1212453446=9, srv1421344993=23, srv1253650368=14, srv712158209=83, srv648937576=80, srv1220279209=10, srv329508097=66, srv880710558=94, srv787812609=87, srv1259077383=15, srv361319893=68, srv930074797=96, srv1234413888=12, srv741240449=85, srv1369851591=21, srv1783674610=39, srv981331916=99, srv204849378=49, srv637628560=79, srv1088476612=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-04 09:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-04 09:53:17,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-04 09:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-04 09:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-04 09:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-04 09:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-04 09:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-04 09:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-04 09:53:17,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-04 09:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-04 09:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-04 09:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-04 09:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-04 09:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-04 09:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-04 09:53:17,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-04 09:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-04 09:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-04 09:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-04 09:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-04 09:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-04 09:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-04 09:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-04 09:53:17,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-04 09:53:17,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-04 09:53:17,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-04 09:53:17,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-04 09:53:17,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-04 09:53:17,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-04 09:53:17,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-04 09:53:17,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-04 09:53:17,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-04 09:53:17,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-04 09:53:17,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-04 09:53:17,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-04 09:53:17,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-04 09:53:17,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-04 09:53:17,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-04 09:53:17,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-04 09:53:17,076 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-04 09:53:17,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-04 09:53:17,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-04 09:53:17,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-04 09:53:17,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-04 09:53:17,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-04 09:53:17,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-04 09:53:17,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-04 09:53:17,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-04 09:53:17,077 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-04 09:53:17,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-04 09:53:17,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-04 09:53:17,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-04 09:53:17,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-04 09:53:17,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-04 09:53:17,078 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-04 09:53:17,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-04 09:53:17,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-04 09:53:17,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-04 09:53:17,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-04 09:53:17,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-04 09:53:17,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-04 09:53:17,079 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-04 09:53:17,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-04 09:53:17,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-04 09:53:17,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-04 09:53:17,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-04 09:53:17,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-04 09:53:17,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-04 09:53:17,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-04 09:53:17,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-04 09:53:17,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-04 09:53:17,080 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-04 09:53:17,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-04 09:53:17,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-04 09:53:17,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-04 09:53:17,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-04 09:53:17,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-04 09:53:17,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-04 09:53:17,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-04 09:53:17,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-04 09:53:17,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-04 09:53:17,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-04 09:53:17,081 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-04 09:53:17,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-04 09:53:17,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-04 09:53:17,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-04 09:53:17,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-04 09:53:17,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-04 09:53:17,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-04 09:53:17,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-04 09:53:17,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-04 09:53:17,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-04 09:53:17,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-04 09:53:17,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-04 09:53:17,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-04 09:53:17,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-04 09:53:17,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-04 09:53:17,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-04 09:53:17,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-04 09:53:17,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-04 09:53:17,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-04 09:53:17,389 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-04 09:53:17,390 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-04 09:53:17,390 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-04 09:53:17,848 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 739 ms to try 5000 different iterations.  Found a solution that moves 4683 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.1788432317068276. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039075869236053926, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054976783182661346, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5203333333333333, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039075869236053926, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2405, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-04 09:53:17,860 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv521494836:81 , srv780460827:81 , srv1158880794:82 , srv1857800905:82 , srv1869041242:82 , srv981331916:82 , srv1392214657:83 , srv1552114119:83 , srv1783674610:83 , srv294029025:83 , srv741240449:83 , srv877268363:83 , srv1064489619:84 , srv1088476612:84 , srv1124331355:84 , srv1609009682:84 , srv298796407:84 , srv328719532:84 , srv344883115:84 , srv361319893:84 , srv382148028:84 , srv397640445:84 , srv463020622:84 , srv705684186:84 , srv712158209:84 , srv861470529:84 , srv1110505602:85 , srv1259077383:85 , srv1369851591:85 , srv1472492355:85 , srv1516624642:85 , srv1705873468:85 , srv1827083822:85 , srv1886799389:85 , srv2125726678:85 , srv228553675:85 , srv441935816:85 , srv47968199:85 , srv637628560:85 , srv862419198:85 , srv1220279209:86 , srv1237409763:86 , srv1421344993:86 , srv204849378:86 , srv235621702:86 , srv325384829:86 , srv329338269:86 , srv551143944:86 , srv799026841:86 , srv880710558:86 , srv88463185:86 , srv1168034411:87 , srv1309440237:87 , srv1321336267:87 , srv1431431150:87 , srv1723066229:87 , srv1760425567:87 , srv2051205482:87 , srv2087960017:87 , srv2119590923:87 , srv234040835:87 , srv289644545:87 , srv329508097:87 , srv614604855:87 , srv665853997:87 , srv842679828:87 , srv1133088850:88 , srv1253650368:88 , srv1328172477:88 , srv1468772263:88 , srv1729291934:88 , srv174626398:88 , srv1767910349:88 , srv1854986208:88 , srv1889265964:88 , srv1891675792:88 , srv1892557721:88 , srv2080382958:88 , srv967968061:88 , srv1140987548:89 , srv1212453446:89 , srv1229146344:89 , srv134874889:89 , srv572976011:89 , srv852345879:89 , srv1333672068:90 , srv1540446986:90 , srv1953165476:90 , srv2122106230:90 , srv279375121:90 , srv561486382:90 , srv73145001:90 , srv1682358881:91 , srv787812609:91 , srv951082267:91 , srv1051640485:92 , srv1753083489:92 , srv1234413888:95 , srv930074797:98 , srv648937576:437 }
2024-01-04 09:53:17,864 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-04 09:53:17,865 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1133088850=5, srv235621702=58, srv1886799389=44, srv705684186=82, srv1237409763=13, srv1723066229=33, srv1891675792=46, srv1760425567=37, srv1753083489=36, srv1431431150=24, srv1682358881=31, srv294029025=61, srv1889265964=45, srv328719532=64, srv1767910349=38, srv279375121=59, srv1854986208=41, srv1552114119=29, srv551143944=75, srv397640445=70, srv174626398=35, srv88463185=95, srv1309440237=16, srv325384829=63, srv47968199=73, srv1321336267=17, srv1953165476=48, srv1472492355=26, srv1869041242=43, srv951082267=97, srv1110505602=3, srv861470529=91, srv2122106230=54, srv614604855=78, srv298796407=62, srv2080382958=51, srv1468772263=25, srv572976011=77, srv134874889=20, srv2125726678=55, srv799026841=88, srv1729291934=34, srv1158880794=7, srv1392214657=22, srv1229146344=11, srv463020622=72, srv2051205482=50, srv862419198=92, srv1064489619=1, srv1333672068=19, srv2087960017=52, srv1540446986=28, srv234040835=57, srv842679828=89, srv967968061=98, srv1609009682=30, srv877268363=93, srv1168034411=8, srv780460827=86, srv1124331355=4, srv1857800905=42, srv1892557721=47, srv329338269=65, srv2119590923=53, srv382148028=69, srv73145001=84, srv1827083822=40, srv289644545=60, srv441935816=71, srv521494836=74, srv228553675=56, srv852345879=90, srv1140987548=6, srv1705873468=32, srv561486382=76, srv665853997=81, srv1516624642=27, srv1051640485=0, srv344883115=67, srv1328172477=18, srv1212453446=9, srv1421344993=23, srv1253650368=14, srv712158209=83, srv648937576=80, srv1220279209=10, srv329508097=66, srv880710558=94, srv787812609=87, srv1259077383=15, srv361319893=68, srv930074797=96, srv1234413888=12, srv741240449=85, srv1369851591=21, srv1783674610=39, srv981331916=99, srv204849378=49, srv637628560=79, srv1088476612=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-04 09:53:17,882 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-04 09:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-04 09:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-04 09:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-04 09:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-04 09:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-04 09:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-04 09:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-04 09:53:17,883 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-04 09:53:17,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-04 09:53:17,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-04 09:53:17,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-04 09:53:17,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-04 09:53:17,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-04 09:53:17,884 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-04 09:53:17,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-04 09:53:17,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-04 09:53:17,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-04 09:53:17,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-04 09:53:17,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-04 09:53:17,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-04 09:53:17,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-04 09:53:17,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-04 09:53:17,885 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-04 09:53:17,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-04 09:53:17,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-04 09:53:17,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-04 09:53:17,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-04 09:53:17,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-04 09:53:17,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-04 09:53:17,886 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-04 09:53:17,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-04 09:53:17,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-04 09:53:17,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-04 09:53:17,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-04 09:53:17,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-04 09:53:17,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-04 09:53:17,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-04 09:53:17,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-04 09:53:17,887 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-04 09:53:17,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-04 09:53:17,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-04 09:53:17,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-04 09:53:17,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-04 09:53:17,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-04 09:53:17,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-04 09:53:17,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-04 09:53:17,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-04 09:53:17,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-04 09:53:17,888 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-04 09:53:17,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-04 09:53:17,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-04 09:53:17,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-04 09:53:17,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-04 09:53:17,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-04 09:53:17,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-04 09:53:17,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-04 09:53:17,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-04 09:53:17,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-04 09:53:17,889 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-04 09:53:17,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-04 09:53:17,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-04 09:53:17,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-04 09:53:17,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-04 09:53:17,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-04 09:53:17,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-04 09:53:17,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-04 09:53:17,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-04 09:53:17,890 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-04 09:53:17,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-04 09:53:17,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-04 09:53:17,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-04 09:53:17,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-04 09:53:17,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-04 09:53:17,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-04 09:53:17,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-04 09:53:17,891 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-04 09:53:17,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-04 09:53:17,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-04 09:53:17,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-04 09:53:17,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-04 09:53:17,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-04 09:53:17,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-04 09:53:17,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-04 09:53:17,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-04 09:53:17,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-04 09:53:17,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-04 09:53:17,892 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-04 09:53:17,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-04 09:53:17,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-04 09:53:17,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-04 09:53:17,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-04 09:53:17,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-04 09:53:17,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-04 09:53:17,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-04 09:53:17,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-04 09:53:17,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-04 09:53:17,893 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-04 09:53:17,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-04 09:53:17,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-04 09:53:17,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-04 09:53:18,099 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039075869236053926, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054976783182661346, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039075869236053926, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2405, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-04 09:53:18,100 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-04 09:53:18,100 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.17881044063997006, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.039075869236053926, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.054976783182661346, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.039075869236053926, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17408333333333334, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2405, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-04 09:53:18,414 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 493 ms to try 5000 different iterations.  Found a solution that moves 2609 regions; Going from a computed imbalance of 0.17881044063997006 to a new imbalance of 0.017865988440115835. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02238771152073282); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03932083348719793, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.28988888888888886, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02238771152073282); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01475); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04758333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-04 09:53:18,429 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv930074797:73 , srv1051640485:78 , srv1753083489:79 , srv1234413888:80 , srv73145001:80 , srv561486382:81 , srv951082267:81 , srv1140987548:82 , srv1229146344:82 , srv134874889:82 , srv1468772263:82 , srv174626398:82 , srv1767910349:82 , srv1854986208:82 , srv1889265964:82 , srv279375121:82 , srv852345879:82 , srv1212453446:83 , srv1309440237:83 , srv1333672068:83 , srv1682358881:83 , srv1729291934:83 , srv1953165476:83 , srv2122106230:83 , srv787812609:83 , srv1540446986:84 , srv1891675792:84 , srv2080382958:84 , srv2119590923:84 , srv289644545:84 , srv329338269:84 , srv1133088850:85 , srv1220279209:85 , srv1253650368:85 , srv1321336267:85 , srv1723066229:85 , srv1886799389:85 , srv1892557721:85 , srv234040835:85 , srv235621702:85 , srv572976011:85 , srv614604855:85 , srv665853997:85 , srv967968061:85 , srv1064489619:86 , srv1124331355:86 , srv1168034411:86 , srv1237409763:86 , srv1328172477:86 , srv1369851591:86 , srv1421344993:86 , srv1472492355:86 , srv1516624642:86 , srv1705873468:86 , srv1827083822:86 , srv2051205482:86 , srv228553675:86 , srv329508097:86 , srv344883115:86 , srv551143944:86 , srv705684186:86 , srv880710558:86 , srv88463185:86 , srv1110505602:87 , srv1609009682:87 , srv2087960017:87 , srv2125726678:87 , srv328719532:87 , srv361319893:87 , srv637628560:87 , srv799026841:87 , srv842679828:87 , srv862419198:87 , srv1088476612:88 , srv1259077383:88 , srv1431431150:88 , srv1552114119:88 , srv1760425567:88 , srv1783674610:88 , srv294029025:88 , srv325384829:88 , srv382148028:88 , srv861470529:88 , srv204849378:89 , srv298796407:89 , srv397640445:89 , srv47968199:89 , srv521494836:89 , srv741240449:89 , srv877268363:89 , srv1158880794:90 , srv1392214657:90 , srv1857800905:90 , srv441935816:90 , srv463020622:90 , srv712158209:90 , srv780460827:90 , srv1869041242:91 , srv981331916:91 , srv648937576:536 }
2024-01-04 09:53:18,436 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-04 09:53:18,437 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1133088850=5, srv235621702=58, srv1886799389=44, srv705684186=82, srv1237409763=13, srv1723066229=33, srv1891675792=46, srv1760425567=37, srv1753083489=36, srv1431431150=24, srv1682358881=31, srv294029025=61, srv1889265964=45, srv328719532=64, srv1767910349=38, srv279375121=59, srv1854986208=41, srv1552114119=29, srv551143944=75, srv397640445=70, srv174626398=35, srv88463185=95, srv1309440237=16, srv325384829=63, srv47968199=73, srv1321336267=17, srv1953165476=48, srv1472492355=26, srv1869041242=43, srv951082267=97, srv1110505602=3, srv861470529=91, srv2122106230=54, srv614604855=78, srv298796407=62, srv2080382958=51, srv1468772263=25, srv572976011=77, srv134874889=20, srv2125726678=55, srv799026841=88, srv1729291934=34, srv1158880794=7, srv1392214657=22, srv1229146344=11, srv463020622=72, srv2051205482=50, srv862419198=92, srv1064489619=1, srv1333672068=19, srv2087960017=52, srv1540446986=28, srv234040835=57, srv842679828=89, srv967968061=98, srv1609009682=30, srv877268363=93, srv1168034411=8, srv780460827=86, srv1124331355=4, srv1857800905=42, srv1892557721=47, srv329338269=65, srv2119590923=53, srv382148028=69, srv73145001=84, srv1827083822=40, srv289644545=60, srv441935816=71, srv521494836=74, srv228553675=56, srv852345879=90, srv1140987548=6, srv1705873468=32, srv561486382=76, srv665853997=81, srv1516624642=27, srv1051640485=0, srv344883115=67, srv1328172477=18, srv1212453446=9, srv1421344993=23, srv1253650368=14, srv712158209=83, srv648937576=80, srv1220279209=10, srv329508097=66, srv880710558=94, srv787812609=87, srv1259077383=15, srv361319893=68, srv930074797=96, srv1234413888=12, srv741240449=85, srv1369851591=21, srv1783674610=39, srv981331916=99, srv204849378=49, srv637628560=79, srv1088476612=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-04 09:53:18,460 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-04 09:53:18,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-04 09:53:18,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-04 09:53:18,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-04 09:53:18,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-04 09:53:18,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-04 09:53:18,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-04 09:53:18,461 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-04 09:53:18,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-04 09:53:18,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-04 09:53:18,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-04 09:53:18,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-04 09:53:18,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-04 09:53:18,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-04 09:53:18,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-04 09:53:18,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-04 09:53:18,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-04 09:53:18,462 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-04 09:53:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-04 09:53:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-04 09:53:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-04 09:53:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-04 09:53:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-04 09:53:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-04 09:53:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-04 09:53:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-04 09:53:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-04 09:53:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-04 09:53:18,463 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-04 09:53:18,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-04 09:53:18,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-04 09:53:18,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-04 09:53:18,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-04 09:53:18,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-04 09:53:18,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-04 09:53:18,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-04 09:53:18,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-04 09:53:18,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-04 09:53:18,464 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-04 09:53:18,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-04 09:53:18,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-04 09:53:18,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-04 09:53:18,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-04 09:53:18,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-04 09:53:18,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-04 09:53:18,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-04 09:53:18,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-04 09:53:18,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-04 09:53:18,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-04 09:53:18,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-04 09:53:18,465 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-04 09:53:18,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-04 09:53:18,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-04 09:53:18,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-04 09:53:18,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-04 09:53:18,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-04 09:53:18,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-04 09:53:18,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-04 09:53:18,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-04 09:53:18,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-04 09:53:18,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-04 09:53:18,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-04 09:53:18,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-04 09:53:18,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-04 09:53:18,466 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-04 09:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-04 09:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-04 09:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-04 09:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-04 09:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-04 09:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-04 09:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-04 09:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-04 09:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-04 09:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-04 09:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-04 09:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-04 09:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-04 09:53:18,467 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-04 09:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-04 09:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-04 09:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-04 09:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-04 09:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-04 09:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-04 09:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-04 09:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-04 09:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-04 09:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-04 09:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-04 09:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-04 09:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-04 09:53:18,468 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-04 09:53:18,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-04 09:53:18,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-04 09:53:18,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-04 09:53:18,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-04 09:53:18,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-04 09:53:18,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-04 09:53:18,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-04 09:53:18,469 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-04 09:53:18,622 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02238771152073282); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03932083348719793, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02238771152073282); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01475); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04758333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-04 09:53:18,623 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-04 09:53:18,623 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.017847719831653036, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.02238771152073282); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03932083348719793, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.02238771152073282); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.01475); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04758333333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-04 09:53:19,009 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 522 ms to try 5000 different iterations.  Found a solution that moves 1025 regions; Going from a computed imbalance of 0.017847719831653036 to a new imbalance of 0.0011825728677127489. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00835817472142689); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010508934790820164); MoveCostFunction : (multiplier=7.0, imbalance=0.11388888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00835817472142689); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=7.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-04 09:53:19,011 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1234413888:82 , srv1431431150:82 , srv1760425567:82 , srv441935816:82 , srv572976011:82 , srv787812609:82 , srv1168034411:83 , srv1253650368:83 , srv1892557721:83 , srv204849378:83 , srv2087960017:83 , srv325384829:83 , srv463020622:83 , srv712158209:83 , srv1237409763:84 , srv1259077383:84 , srv1328172477:84 , srv1392214657:84 , srv1540446986:84 , srv1682358881:84 , srv1953165476:84 , srv2051205482:84 , srv2125726678:84 , srv47968199:84 , srv551143944:84 , srv741240449:84 , srv799026841:84 , srv842679828:84 , srv1110505602:85 , srv1133088850:85 , srv1212453446:85 , srv1229146344:85 , srv1333672068:85 , srv1552114119:85 , srv1705873468:85 , srv1723066229:85 , srv1827083822:85 , srv1857800905:85 , srv2080382958:85 , srv2122106230:85 , srv228553675:85 , srv234040835:85 , srv298796407:85 , srv329508097:85 , srv382148028:85 , srv561486382:85 , srv614604855:85 , srv877268363:85 , srv880710558:85 , srv930074797:85 , srv967968061:85 , srv1088476612:86 , srv1140987548:86 , srv1158880794:86 , srv1220279209:86 , srv1321336267:86 , srv134874889:86 , srv1369851591:86 , srv1472492355:86 , srv1609009682:86 , srv1753083489:86 , srv1783674610:86 , srv1854986208:86 , srv1869041242:86 , srv1886799389:86 , srv1891675792:86 , srv235621702:86 , srv279375121:86 , srv289644545:86 , srv328719532:86 , srv344883115:86 , srv397640445:86 , srv637628560:86 , srv665853997:86 , srv780460827:86 , srv861470529:86 , srv862419198:86 , srv88463185:86 , srv981331916:86 , srv1064489619:87 , srv1421344993:87 , srv1468772263:87 , srv1516624642:87 , srv1729291934:87 , srv174626398:87 , srv1767910349:87 , srv1889265964:87 , srv294029025:87 , srv361319893:87 , srv521494836:87 , srv705684186:87 , srv73145001:87 , srv852345879:87 , srv951082267:87 , srv1051640485:88 , srv1309440237:88 , srv2119590923:88 , srv329338269:88 , srv1124331355:89 , srv648937576:559 }
2024-01-04 09:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-04 09:53:19,015 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1133088850=5, srv235621702=58, srv1886799389=44, srv705684186=82, srv1237409763=13, srv1723066229=33, srv1891675792=46, srv1760425567=37, srv1753083489=36, srv1431431150=24, srv1682358881=31, srv294029025=61, srv1889265964=45, srv328719532=64, srv1767910349=38, srv279375121=59, srv1854986208=41, srv1552114119=29, srv551143944=75, srv397640445=70, srv174626398=35, srv88463185=95, srv1309440237=16, srv325384829=63, srv47968199=73, srv1321336267=17, srv1953165476=48, srv1472492355=26, srv1869041242=43, srv951082267=97, srv1110505602=3, srv861470529=91, srv2122106230=54, srv614604855=78, srv298796407=62, srv2080382958=51, srv1468772263=25, srv572976011=77, srv134874889=20, srv2125726678=55, srv799026841=88, srv1729291934=34, srv1158880794=7, srv1392214657=22, srv1229146344=11, srv463020622=72, srv2051205482=50, srv862419198=92, srv1064489619=1, srv1333672068=19, srv2087960017=52, srv1540446986=28, srv234040835=57, srv842679828=89, srv967968061=98, srv1609009682=30, srv877268363=93, srv1168034411=8, srv780460827=86, srv1124331355=4, srv1857800905=42, srv1892557721=47, srv329338269=65, srv2119590923=53, srv382148028=69, srv73145001=84, srv1827083822=40, srv289644545=60, srv441935816=71, srv521494836=74, srv228553675=56, srv852345879=90, srv1140987548=6, srv1705873468=32, srv561486382=76, srv665853997=81, srv1516624642=27, srv1051640485=0, srv344883115=67, srv1328172477=18, srv1212453446=9, srv1421344993=23, srv1253650368=14, srv712158209=83, srv648937576=80, srv1220279209=10, srv329508097=66, srv880710558=94, srv787812609=87, srv1259077383=15, srv361319893=68, srv930074797=96, srv1234413888=12, srv741240449=85, srv1369851591=21, srv1783674610=39, srv981331916=99, srv204849378=49, srv637628560=79, srv1088476612=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-04 09:53:19,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-04 09:53:19,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-04 09:53:19,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-04 09:53:19,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-04 09:53:19,028 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-04 09:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-04 09:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-04 09:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-04 09:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-04 09:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-04 09:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-04 09:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-04 09:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-04 09:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-04 09:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-04 09:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-04 09:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-04 09:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-04 09:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-04 09:53:19,029 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-04 09:53:19,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-04 09:53:19,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-04 09:53:19,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-04 09:53:19,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-04 09:53:19,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-04 09:53:19,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-04 09:53:19,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-04 09:53:19,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-04 09:53:19,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-04 09:53:19,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-04 09:53:19,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-04 09:53:19,030 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-04 09:53:19,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-04 09:53:19,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-04 09:53:19,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-04 09:53:19,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-04 09:53:19,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-04 09:53:19,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-04 09:53:19,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-04 09:53:19,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-04 09:53:19,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-04 09:53:19,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-04 09:53:19,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-04 09:53:19,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-04 09:53:19,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-04 09:53:19,031 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-04 09:53:19,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-04 09:53:19,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-04 09:53:19,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-04 09:53:19,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-04 09:53:19,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-04 09:53:19,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-04 09:53:19,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-04 09:53:19,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-04 09:53:19,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-04 09:53:19,032 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-04 09:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-04 09:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-04 09:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-04 09:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-04 09:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-04 09:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-04 09:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-04 09:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-04 09:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-04 09:53:19,033 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-04 09:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-04 09:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-04 09:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-04 09:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-04 09:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-04 09:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-04 09:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-04 09:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-04 09:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-04 09:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-04 09:53:19,034 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-04 09:53:19,035 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-04 09:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-04 09:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-04 09:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-04 09:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-04 09:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-04 09:53:19,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-04 09:53:19,167 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00835817472142689); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010508934790820164); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00835817472142689); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=7.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-04 09:53:19,167 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-01-04 09:53:19,168 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.0011753956643113048, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00835817472142689); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.010508934790820164); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.00835817472142689); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=7.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004583333333333333); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-01-04 09:53:19,422 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 365 ms to try 5000 different iterations.  Found a solution that moves 289 regions; Going from a computed imbalance of 0.0011753956643113048 to a new imbalance of 2.033160386581883E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.03211111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-04 09:53:19,423 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1124331355:83 , srv1421344993:83 , srv1540446986:83 , srv1869041242:83 , srv2119590923:83 , srv397640445:83 , srv951082267:83 , srv1051640485:84 , srv1088476612:84 , srv1158880794:84 , srv1309440237:84 , srv1321336267:84 , srv1333672068:84 , srv1516624642:84 , srv1682358881:84 , srv1729291934:84 , srv1891675792:84 , srv204849378:84 , srv2122106230:84 , srv279375121:84 , srv294029025:84 , srv298796407:84 , srv328719532:84 , srv329338269:84 , srv329508097:84 , srv361319893:84 , srv47968199:84 , srv665853997:84 , srv799026841:84 , srv842679828:84 , srv861470529:84 , srv862419198:84 , srv88463185:84 , srv967968061:84 , srv981331916:84 , srv1064489619:85 , srv1110505602:85 , srv1133088850:85 , srv1140987548:85 , srv1212453446:85 , srv1220279209:85 , srv1234413888:85 , srv1259077383:85 , srv1328172477:85 , srv134874889:85 , srv1369851591:85 , srv1392214657:85 , srv1431431150:85 , srv1468772263:85 , srv1472492355:85 , srv1609009682:85 , srv1705873468:85 , srv1723066229:85 , srv174626398:85 , srv1753083489:85 , srv1760425567:85 , srv1767910349:85 , srv1783674610:85 , srv1827083822:85 , srv1857800905:85 , srv1886799389:85 , srv1889265964:85 , srv1953165476:85 , srv2051205482:85 , srv2080382958:85 , srv2087960017:85 , srv234040835:85 , srv235621702:85 , srv289644545:85 , srv325384829:85 , srv382148028:85 , srv441935816:85 , srv463020622:85 , srv521494836:85 , srv614604855:85 , srv637628560:85 , srv712158209:85 , srv73145001:85 , srv741240449:85 , srv780460827:85 , srv852345879:85 , srv877268363:85 , srv880710558:85 , srv1168034411:86 , srv1229146344:86 , srv1237409763:86 , srv1253650368:86 , srv1552114119:86 , srv1854986208:86 , srv1892557721:86 , srv2125726678:86 , srv228553675:86 , srv344883115:86 , srv551143944:86 , srv561486382:86 , srv572976011:86 , srv705684186:86 , srv930074797:86 , srv787812609:87 , srv648937576:610 }
2024-01-04 09:53:19,426 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-04 09:53:19,427 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1133088850=5, srv235621702=58, srv1886799389=44, srv705684186=82, srv1237409763=13, srv1723066229=33, srv1891675792=46, srv1760425567=37, srv1753083489=36, srv1431431150=24, srv1682358881=31, srv294029025=61, srv1889265964=45, srv328719532=64, srv1767910349=38, srv279375121=59, srv1854986208=41, srv1552114119=29, srv551143944=75, srv397640445=70, srv174626398=35, srv88463185=95, srv1309440237=16, srv325384829=63, srv47968199=73, srv1321336267=17, srv1953165476=48, srv1472492355=26, srv1869041242=43, srv951082267=97, srv1110505602=3, srv861470529=91, srv2122106230=54, srv614604855=78, srv298796407=62, srv2080382958=51, srv1468772263=25, srv572976011=77, srv134874889=20, srv2125726678=55, srv799026841=88, srv1729291934=34, srv1158880794=7, srv1392214657=22, srv1229146344=11, srv463020622=72, srv2051205482=50, srv862419198=92, srv1064489619=1, srv1333672068=19, srv2087960017=52, srv1540446986=28, srv234040835=57, srv842679828=89, srv967968061=98, srv1609009682=30, srv877268363=93, srv1168034411=8, srv780460827=86, srv1124331355=4, srv1857800905=42, srv1892557721=47, srv329338269=65, srv2119590923=53, srv382148028=69, srv73145001=84, srv1827083822=40, srv289644545=60, srv441935816=71, srv521494836=74, srv228553675=56, srv852345879=90, srv1140987548=6, srv1705873468=32, srv561486382=76, srv665853997=81, srv1516624642=27, srv1051640485=0, srv344883115=67, srv1328172477=18, srv1212453446=9, srv1421344993=23, srv1253650368=14, srv712158209=83, srv648937576=80, srv1220279209=10, srv329508097=66, srv880710558=94, srv787812609=87, srv1259077383=15, srv361319893=68, srv930074797=96, srv1234413888=12, srv741240449=85, srv1369851591=21, srv1783674610=39, srv981331916=99, srv204849378=49, srv637628560=79, srv1088476612=2} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-04 09:53:19,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-04 09:53:19,438 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-04 09:53:19,439 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-01-04 09:53:19,440 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-01-04 09:53:19,441 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-01-04 09:53:19,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-01-04 09:53:19,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-01-04 09:53:19,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-01-04 09:53:19,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-01-04 09:53:19,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-01-04 09:53:19,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-01-04 09:53:19,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-01-04 09:53:19,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-01-04 09:53:19,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-01-04 09:53:19,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-01-04 09:53:19,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-01-04 09:53:19,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-01-04 09:53:19,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-01-04 09:53:19,442 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-01-04 09:53:19,443 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-01-04 09:53:19,557 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=1.8307982614094552E-5 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.178341294946416E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0019534423844256277); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.178341294946416E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=8.333333333333333E-5); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-04 09:53:19,557 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1124331355:83 , srv1421344993:83 , srv1540446986:83 , srv1869041242:83 , srv2119590923:83 , srv397640445:83 , srv951082267:83 , srv1051640485:84 , srv1088476612:84 , srv1158880794:84 , srv1309440237:84 , srv1321336267:84 , srv1333672068:84 , srv1516624642:84 , srv1682358881:84 , srv1729291934:84 , srv1891675792:84 , srv204849378:84 , srv2122106230:84 , srv279375121:84 , srv294029025:84 , srv298796407:84 , srv328719532:84 , srv329338269:84 , srv329508097:84 , srv361319893:84 , srv47968199:84 , srv665853997:84 , srv799026841:84 , srv842679828:84 , srv861470529:84 , srv862419198:84 , srv88463185:84 , srv967968061:84 , srv981331916:84 , srv1064489619:85 , srv1110505602:85 , srv1133088850:85 , srv1140987548:85 , srv1212453446:85 , srv1220279209:85 , srv1234413888:85 , srv1259077383:85 , srv1328172477:85 , srv134874889:85 , srv1369851591:85 , srv1392214657:85 , srv1431431150:85 , srv1468772263:85 , srv1472492355:85 , srv1609009682:85 , srv1705873468:85 , srv1723066229:85 , srv174626398:85 , srv1753083489:85 , srv1760425567:85 , srv1767910349:85 , srv1783674610:85 , srv1827083822:85 , srv1857800905:85 , srv1886799389:85 , srv1889265964:85 , srv1953165476:85 , srv2051205482:85 , srv2080382958:85 , srv2087960017:85 , srv234040835:85 , srv235621702:85 , srv289644545:85 , srv325384829:85 , srv382148028:85 , srv441935816:85 , srv463020622:85 , srv521494836:85 , srv614604855:85 , srv637628560:85 , srv712158209:85 , srv73145001:85 , srv741240449:85 , srv780460827:85 , srv852345879:85 , srv877268363:85 , srv880710558:85 , srv1168034411:86 , srv1229146344:86 , srv1237409763:86 , srv1253650368:86 , srv1552114119:86 , srv1854986208:86 , srv1892557721:86 , srv2125726678:86 , srv228553675:86 , srv344883115:86 , srv551143944:86 , srv561486382:86 , srv572976011:86 , srv705684186:86 , srv930074797:86 , srv787812609:87 , srv648937576:610 }
2024-01-04 09:53:19,591 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=304 (was 304), ProcessCount=168 (was 168), AvailableMemoryMB=10771 (was 11376)
2024-01-04 09:53:19,599 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=304, ProcessCount=168, AvailableMemoryMB=10771
2024-01-04 09:53:19,599 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-01-04 09:53:19,600 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-01-04 09:53:19,608 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1030105672:84 , srv114736954:84 , srv1189475612:84 , srv1294414930:84 , srv1301219124:84 , srv1403380228:84 , srv1529670845:84 , srv155811251:84 , srv1605769371:84 , srv1617050174:84 , srv1682520868:84 , srv1694317174:84 , srv1718721172:84 , srv172494467:84 , srv1892479792:84 , srv1913782557:84 , srv1997348112:84 , srv2002573116:84 , srv2033151166:84 , srv278259277:84 , srv396942527:84 , srv418756733:84 , srv442480647:84 , srv563071449:84 , srv608998415:84 , srv613134594:84 , srv799332397:84 , srv803828790:84 , srv898949012:84 , srv957242754:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-01-04 09:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-04 09:53:19,609 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv114736954=1, srv803828790=27, srv1605769371=8, srv563071449=23, srv1301219124=4, srv155811251=7, srv278259277=19, srv1892479792=14, srv396942527=20, srv2002573116=17, srv172494467=13, srv613134594=25, srv1694317174=11, srv1718721172=12, srv1682520868=10, srv898949012=28, srv608998415=24, srv1030105672=0, srv1403380228=5, srv442480647=22, srv1913782557=15, srv1294414930=3, srv1529670845=6, srv1189475612=2, srv418756733=21, srv799332397=26, srv1997348112=16, srv957242754=29, srv2033151166=18, srv1617050174=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-04 09:53:19,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-04 09:53:19,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-04 09:53:19,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-04 09:53:19,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-04 09:53:19,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-04 09:53:19,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-04 09:53:19,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-04 09:53:19,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-04 09:53:19,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-04 09:53:19,618 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-04 09:53:19,619 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-04 09:53:19,634 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0856710021931772, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-04 09:53:19,635 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912462989794775, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.0856710021931772, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-01-04 09:55:19,623 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 5393166 different iterations.  Found a solution that moves 1865 regions; Going from a computed imbalance of 0.9912462989794775 to a new imbalance of 4.353003038599517E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6907407407407408, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-04 09:55:19,630 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1030105672:90 , srv114736954:90 , srv1189475612:90 , srv1294414930:90 , srv1301219124:90 , srv1403380228:90 , srv1529670845:90 , srv155811251:90 , srv1605769371:90 , srv1617050174:90 , srv1682520868:90 , srv1694317174:90 , srv1718721172:90 , srv172494467:90 , srv1892479792:90 , srv1913782557:90 , srv1997348112:90 , srv2002573116:90 , srv2033151166:90 , srv278259277:90 , srv396942527:90 , srv418756733:90 , srv442480647:90 , srv563071449:90 , srv608998415:90 , srv613134594:90 , srv799332397:90 , srv803828790:90 , srv898949012:90 , srv957242754:90 }
2024-01-04 09:55:19,632 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-01-04 09:55:19,633 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv114736954=1, srv803828790=27, srv1605769371=8, srv563071449=23, srv1301219124=4, srv155811251=7, srv278259277=19, srv1892479792=14, srv396942527=20, srv2002573116=17, srv172494467=13, srv613134594=25, srv1694317174=11, srv1718721172=12, srv1682520868=10, srv898949012=28, srv608998415=24, srv1030105672=0, srv1403380228=5, srv442480647=22, srv1913782557=15, srv1294414930=3, srv1529670845=6, srv1189475612=2, srv418756733=21, srv799332397=26, srv1997348112=16, srv957242754=29, srv2033151166=18, srv1617050174=9} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-01-04 09:55:19,638 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-01-04 09:55:19,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-01-04 09:55:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-01-04 09:55:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-01-04 09:55:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-01-04 09:55:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-01-04 09:55:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-01-04 09:55:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-01-04 09:55:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-01-04 09:55:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-01-04 09:55:19,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-01-04 09:55:19,652 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-01-04 09:55:19,652 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1030105672:90 , srv114736954:90 , srv1189475612:90 , srv1294414930:90 , srv1301219124:90 , srv1403380228:90 , srv1529670845:90 , srv155811251:90 , srv1605769371:90 , srv1617050174:90 , srv1682520868:90 , srv1694317174:90 , srv1718721172:90 , srv172494467:90 , srv1892479792:90 , srv1913782557:90 , srv1997348112:90 , srv2002573116:90 , srv2033151166:90 , srv278259277:90 , srv396942527:90 , srv418756733:90 , srv442480647:90 , srv563071449:90 , srv608998415:90 , srv613134594:90 , srv799332397:90 , srv803828790:90 , srv898949012:90 , srv957242754:90 }
2024-01-04 09:55:19,665 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=185 (was 304), ProcessCount=168 (was 168), AvailableMemoryMB=10901 (was 10771) - AvailableMemoryMB LEAK? -