2024-03-22 09:55:05,518 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2024-03-22 09:55:05,765 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2024-03-22 09:55:05,872 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2024-03-22 09:55:05,926 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-22 09:55:05,932 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-22 09:55:05,945 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=149, ProcessCount=177, AvailableMemoryMB=6904
2024-03-22 09:55:05,947 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-22 09:55:05,947 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2024-03-22 09:55:06,052 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1012159117:84 , srv1031548325:84 , srv1053850981:84 , srv1067270490:84 , srv1067457920:84 , srv1090509916:84 , srv1098210209:84 , srv1101128643:84 , srv1146163406:84 , srv116102831:84 , srv1178156025:84 , srv1190084332:84 , srv119610280:84 , srv122119235:84 , srv1226995976:84 , srv1238452187:84 , srv1277079662:84 , srv1294899473:84 , srv129600805:84 , srv1299197254:84 , srv1307777370:84 , srv1316361956:84 , srv1326708013:84 , srv1422253031:84 , srv1456715987:84 , srv1493435618:84 , srv1495479971:84 , srv1502286726:84 , srv1513055389:84 , srv1515397877:84 , srv1519752066:84 , srv152209053:84 , srv1524565397:84 , srv1545068430:84 , srv1579877076:84 , srv1608285455:84 , srv1621821130:84 , srv16332595:84 , srv163918372:84 , srv1669983753:84 , srv1730094560:84 , srv1790804953:84 , srv180901439:84 , srv1824398514:84 , srv1837150122:84 , srv1868398866:84 , srv1873200792:84 , srv187400948:84 , srv1883853581:84 , srv1890196359:84 , srv1893078965:84 , srv1903099598:84 , srv1908348747:84 , srv1926432852:84 , srv1945991828:84 , srv1950354868:84 , srv1954864239:84 , srv1966635448:84 , srv1979583766:84 , srv1981134601:84 , srv2014404518:84 , srv2044744614:84 , srv2087627228:84 , srv210945548:84 , srv217075551:84 , srv226628313:84 , srv264832822:84 , srv276542643:84 , srv330260884:84 , srv353333731:84 , srv354712443:84 , srv389167650:84 , srv40008358:84 , srv402746380:84 , srv449436364:84 , srv456004216:84 , srv462042567:84 , srv473641108:84 , srv478187054:84 , srv489612940:84 , srv493825201:84 , srv569408956:84 , srv577134704:84 , srv581097806:84 , srv653357094:84 , srv659304182:84 , srv680229292:84 , srv70105925:84 , srv743741212:84 , srv74445196:84 , srv744789257:84 , srv745405510:84 , srv801311144:84 , srv807739984:84 , srv817563877:84 , srv909529841:84 , srv980680761:84 , srv981900791:84 , srv990591052:84 , srv2134029312:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2024-03-22 09:55:06,059 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-22 09:55:06,065 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1067457920=4, srv1307777370=20, srv1519752066=30, srv1824398514=43, srv807739984=94, srv1903099598=51, srv745405510=92, srv226628313=66, srv119610280=12, srv152209053=31, srv449436364=75, srv2087627228=62, srv402746380=74, srv456004216=76, srv743741212=89, srv653357094=85, srv1326708013=22, srv990591052=99, srv1730094560=40, srv1981134601=59, srv122119235=13, srv163918372=38, srv659304182=86, srv1422253031=23, srv489612940=80, srv1908348747=52, srv1031548325=1, srv1868398866=45, srv1090509916=5, srv264832822=67, srv1299197254=19, srv1579877076=34, srv129600805=18, srv981900791=98, srv473641108=78, srv817563877=95, srv187400948=47, srv1294899473=17, srv1883853581=48, srv1873200792=46, srv1495479971=26, srv70105925=88, srv1966635448=57, srv180901439=42, srv1101128643=7, srv1945991828=54, srv1493435618=25, srv1621821130=36, srv1545068430=33, srv330260884=69, srv1893078965=50, srv1954864239=56, srv1456715987=24, srv1669983753=39, srv462042567=77, srv569408956=82, srv1067270490=3, srv1098210209=6, srv74445196=90, srv1146163406=8, srv1502286726=27, srv980680761=97, srv581097806=84, srv1524565397=32, srv2044744614=61, srv276542643=68, srv1790804953=41, srv1238452187=15, srv1190084332=11, srv1837150122=44, srv1316361956=21, srv354712443=71, srv493825201=81, srv680229292=87, srv744789257=91, srv1515397877=29, srv16332595=37, srv353333731=70, srv116102831=9, srv2014404518=60, srv40008358=73, srv1608285455=35, srv1513055389=28, srv1926432852=53, srv1012159117=0, srv1053850981=2, srv210945548=63, srv577134704=83, srv1950354868=55, srv2134029312=64, srv1226995976=14, srv1979583766=58, srv909529841=96, srv1277079662=16, srv389167650=72, srv1890196359=49, srv1178156025=10, srv217075551=65, srv801311144=93, srv478187054=79} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-22 09:55:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-22 09:55:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-22 09:55:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-22 09:55:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-22 09:55:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-22 09:55:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-22 09:55:06,126 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-22 09:55:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-22 09:55:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-22 09:55:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-22 09:55:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-22 09:55:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-22 09:55:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-22 09:55:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-22 09:55:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-22 09:55:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-22 09:55:06,127 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-22 09:55:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-22 09:55:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-22 09:55:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-22 09:55:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-22 09:55:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-22 09:55:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-22 09:55:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-22 09:55:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-22 09:55:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-22 09:55:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-22 09:55:06,128 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-22 09:55:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-22 09:55:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-22 09:55:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-22 09:55:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-22 09:55:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-22 09:55:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-22 09:55:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-22 09:55:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-22 09:55:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-22 09:55:06,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-22 09:55:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-22 09:55:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-22 09:55:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-22 09:55:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-22 09:55:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-22 09:55:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-22 09:55:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-22 09:55:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-22 09:55:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-22 09:55:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-22 09:55:06,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-22 09:55:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-22 09:55:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-22 09:55:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-22 09:55:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-22 09:55:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-22 09:55:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-22 09:55:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-22 09:55:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-22 09:55:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-22 09:55:06,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-22 09:55:06,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-22 09:55:06,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-22 09:55:06,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-22 09:55:06,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-22 09:55:06,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-22 09:55:06,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-22 09:55:06,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-22 09:55:06,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-22 09:55:06,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-22 09:55:06,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-22 09:55:06,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-22 09:55:06,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-22 09:55:06,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-22 09:55:06,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-22 09:55:06,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-22 09:55:06,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-22 09:55:06,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-22 09:55:06,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-22 09:55:06,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-22 09:55:06,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-22 09:55:06,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-22 09:55:06,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-22 09:55:06,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-22 09:55:06,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-22 09:55:06,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-22 09:55:06,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-22 09:55:06,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-22 09:55:06,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-22 09:55:06,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-22 09:55:06,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-22 09:55:06,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-22 09:55:06,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-22 09:55:06,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-22 09:55:06,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-22 09:55:06,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-22 09:55:06,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-22 09:55:06,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-22 09:55:06,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-22 09:55:06,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-22 09:55:06,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-22 09:55:06,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-22 09:55:06,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-22 09:55:06,321 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-22 09:55:06,321 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-22 09:55:06,321 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-22 09:55:06,699 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 545 ms to try 5000 different iterations.  Found a solution that moves 4657 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18733386641875338. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04084315257919688, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05435469824311315, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5174444444444445, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04084315257919688, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1825, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2505833333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-22 09:55:06,707 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1299197254:81 , srv1519752066:81 , srv180901439:81 , srv354712443:81 , srv659304182:82 , srv807739984:82 , srv1012159117:83 , srv116102831:83 , srv122119235:83 , srv1950354868:83 , srv1966635448:83 , srv217075551:83 , srv330260884:83 , srv389167650:83 , srv1098210209:84 , srv1422253031:84 , srv1502286726:84 , srv1868398866:84 , srv1893078965:84 , srv462042567:84 , srv653357094:84 , srv909529841:84 , srv1090509916:85 , srv1294899473:85 , srv129600805:85 , srv1326708013:85 , srv1608285455:85 , srv1790804953:85 , srv187400948:85 , srv1903099598:85 , srv1926432852:85 , srv2014404518:85 , srv2044744614:85 , srv210945548:85 , srv456004216:85 , srv478187054:85 , srv70105925:85 , srv801311144:85 , srv817563877:85 , srv980680761:85 , srv1031548325:86 , srv1067457920:86 , srv1101128643:86 , srv1146163406:86 , srv1178156025:86 , srv1226995976:86 , srv1456715987:86 , srv40008358:86 , srv449436364:86 , srv473641108:86 , srv493825201:86 , srv74445196:86 , srv1053850981:87 , srv1067270490:87 , srv1493435618:87 , srv152209053:87 , srv1524565397:87 , srv1545068430:87 , srv1621821130:87 , srv16332595:87 , srv163918372:87 , srv1824398514:87 , srv1837150122:87 , srv1908348747:87 , srv1945991828:87 , srv1979583766:87 , srv264832822:87 , srv744789257:87 , srv981900791:87 , srv990591052:87 , srv119610280:88 , srv1495479971:88 , srv1513055389:88 , srv1515397877:88 , srv1730094560:88 , srv1873200792:88 , srv1883853581:88 , srv1890196359:88 , srv2087627228:88 , srv402746380:88 , srv680229292:88 , srv743741212:88 , srv489612940:89 , srv569408956:89 , srv1190084332:90 , srv1307777370:90 , srv1579877076:90 , srv226628313:90 , srv353333731:90 , srv577134704:90 , srv1238452187:91 , srv1277079662:91 , srv1316361956:91 , srv1669983753:91 , srv276542643:91 , srv581097806:91 , srv745405510:91 , srv1954864239:92 , srv1981134601:92 , srv2134029312:453 }
2024-03-22 09:55:06,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-22 09:55:06,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1067457920=4, srv1307777370=20, srv1519752066=30, srv1824398514=43, srv807739984=94, srv1903099598=51, srv745405510=92, srv226628313=66, srv119610280=12, srv152209053=31, srv449436364=75, srv2087627228=62, srv402746380=74, srv456004216=76, srv743741212=89, srv653357094=85, srv1326708013=22, srv990591052=99, srv1730094560=40, srv1981134601=59, srv122119235=13, srv163918372=38, srv659304182=86, srv1422253031=23, srv489612940=80, srv1908348747=52, srv1031548325=1, srv1868398866=45, srv1090509916=5, srv264832822=67, srv1299197254=19, srv1579877076=34, srv129600805=18, srv981900791=98, srv473641108=78, srv817563877=95, srv187400948=47, srv1294899473=17, srv1883853581=48, srv1873200792=46, srv1495479971=26, srv70105925=88, srv1966635448=57, srv180901439=42, srv1101128643=7, srv1945991828=54, srv1493435618=25, srv1621821130=36, srv1545068430=33, srv330260884=69, srv1893078965=50, srv1954864239=56, srv1456715987=24, srv1669983753=39, srv462042567=77, srv569408956=82, srv1067270490=3, srv1098210209=6, srv74445196=90, srv1146163406=8, srv1502286726=27, srv980680761=97, srv581097806=84, srv1524565397=32, srv2044744614=61, srv276542643=68, srv1790804953=41, srv1238452187=15, srv1190084332=11, srv1837150122=44, srv1316361956=21, srv354712443=71, srv493825201=81, srv680229292=87, srv744789257=91, srv1515397877=29, srv16332595=37, srv353333731=70, srv116102831=9, srv2014404518=60, srv40008358=73, srv1608285455=35, srv1513055389=28, srv1926432852=53, srv1012159117=0, srv1053850981=2, srv210945548=63, srv577134704=83, srv1950354868=55, srv2134029312=64, srv1226995976=14, srv1979583766=58, srv909529841=96, srv1277079662=16, srv389167650=72, srv1890196359=49, srv1178156025=10, srv217075551=65, srv801311144=93, srv478187054=79} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-22 09:55:06,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-22 09:55:06,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-22 09:55:06,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-22 09:55:06,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-22 09:55:06,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-22 09:55:06,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-22 09:55:06,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-22 09:55:06,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-22 09:55:06,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-22 09:55:06,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-22 09:55:06,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-22 09:55:06,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-22 09:55:06,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-22 09:55:06,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-22 09:55:06,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-22 09:55:06,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-22 09:55:06,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-22 09:55:06,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-22 09:55:06,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-22 09:55:06,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-22 09:55:06,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-22 09:55:06,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-22 09:55:06,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-22 09:55:06,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-22 09:55:06,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-22 09:55:06,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-22 09:55:06,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-22 09:55:06,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-22 09:55:06,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-22 09:55:06,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-22 09:55:06,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-22 09:55:06,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-22 09:55:06,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-22 09:55:06,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-22 09:55:06,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-22 09:55:06,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-22 09:55:06,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-22 09:55:06,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-22 09:55:06,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-22 09:55:06,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-22 09:55:06,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-22 09:55:06,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-22 09:55:06,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-22 09:55:06,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-22 09:55:06,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-22 09:55:06,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-22 09:55:06,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-22 09:55:06,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-22 09:55:06,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-22 09:55:06,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-22 09:55:06,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-22 09:55:06,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-22 09:55:06,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-22 09:55:06,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-22 09:55:06,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-22 09:55:06,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-22 09:55:06,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-22 09:55:06,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-22 09:55:06,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-22 09:55:06,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-22 09:55:06,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-22 09:55:06,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-22 09:55:06,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-22 09:55:06,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-22 09:55:06,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-22 09:55:06,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-22 09:55:06,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-22 09:55:06,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-22 09:55:06,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-22 09:55:06,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-22 09:55:06,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-22 09:55:06,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-22 09:55:06,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-22 09:55:06,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-22 09:55:06,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-22 09:55:06,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-22 09:55:06,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-22 09:55:06,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-22 09:55:06,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-22 09:55:06,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-22 09:55:06,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-22 09:55:06,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-22 09:55:06,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-22 09:55:06,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-22 09:55:06,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-22 09:55:06,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-22 09:55:06,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-22 09:55:06,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-22 09:55:06,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-22 09:55:06,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-22 09:55:06,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-22 09:55:06,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-22 09:55:06,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-22 09:55:06,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-22 09:55:06,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-22 09:55:06,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-22 09:55:06,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-22 09:55:06,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-22 09:55:06,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-22 09:55:06,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-22 09:55:06,740 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-22 09:55:06,842 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04084315257919688, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05435469824311315, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04084315257919688, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1825, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2505833333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-22 09:55:06,843 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-22 09:55:06,843 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.18730125740778703, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04084315257919688, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.05435469824311315, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04084315257919688, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.1825, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.2505833333333333, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-22 09:55:07,038 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 279 ms to try 5000 different iterations.  Found a solution that moves 2662 regions; Going from a computed imbalance of 0.18730125740778703 to a new imbalance of 0.015050799670911323. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023168946885067147); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03466368803572313, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.29577777777777775, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023168946885067147); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04816666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-22 09:55:07,049 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1238452187:80 , srv1579877076:80 , srv1277079662:81 , srv1316361956:81 , srv1981134601:81 , srv1307777370:82 , srv1669983753:82 , srv1954864239:82 , srv276542643:82 , srv353333731:82 , srv489612940:82 , srv680229292:82 , srv1053850981:83 , srv1190084332:83 , srv1621821130:83 , srv1730094560:83 , srv1824398514:83 , srv1883853581:83 , srv1890196359:83 , srv1945991828:83 , srv2087627228:83 , srv226628313:83 , srv577134704:83 , srv581097806:83 , srv745405510:83 , srv981900791:83 , srv119610280:84 , srv1493435618:84 , srv16332595:84 , srv1873200792:84 , srv1979583766:84 , srv40008358:84 , srv402746380:84 , srv473641108:84 , srv569408956:84 , srv743741212:84 , srv1067457920:85 , srv1098210209:85 , srv1178156025:85 , srv1495479971:85 , srv1513055389:85 , srv1545068430:85 , srv1837150122:85 , srv187400948:85 , srv1908348747:85 , srv264832822:85 , srv449436364:85 , srv493825201:85 , srv70105925:85 , srv744789257:85 , srv990591052:85 , srv1067270490:86 , srv1146163406:86 , srv116102831:86 , srv1226995976:86 , srv1608285455:86 , srv210945548:86 , srv801311144:86 , srv817563877:86 , srv1012159117:87 , srv1101128643:87 , srv1294899473:87 , srv129600805:87 , srv1422253031:87 , srv1502286726:87 , srv1515397877:87 , srv152209053:87 , srv1524565397:87 , srv1790804953:87 , srv1903099598:87 , srv1966635448:87 , srv2014404518:87 , srv2044744614:87 , srv456004216:87 , srv478187054:87 , srv74445196:87 , srv1031548325:88 , srv1090509916:88 , srv122119235:88 , srv1326708013:88 , srv1456715987:88 , srv163918372:88 , srv1950354868:88 , srv217075551:88 , srv807739984:88 , srv909529841:88 , srv1868398866:89 , srv1893078965:89 , srv462042567:89 , srv653357094:89 , srv980680761:89 , srv1519752066:90 , srv1926432852:90 , srv330260884:90 , srv389167650:90 , srv659304182:90 , srv180901439:92 , srv354712443:92 , srv1299197254:93 , srv2134029312:527 }
2024-03-22 09:55:07,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-22 09:55:07,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1067457920=4, srv1307777370=20, srv1519752066=30, srv1824398514=43, srv807739984=94, srv1903099598=51, srv745405510=92, srv226628313=66, srv119610280=12, srv152209053=31, srv449436364=75, srv2087627228=62, srv402746380=74, srv456004216=76, srv743741212=89, srv653357094=85, srv1326708013=22, srv990591052=99, srv1730094560=40, srv1981134601=59, srv122119235=13, srv163918372=38, srv659304182=86, srv1422253031=23, srv489612940=80, srv1908348747=52, srv1031548325=1, srv1868398866=45, srv1090509916=5, srv264832822=67, srv1299197254=19, srv1579877076=34, srv129600805=18, srv981900791=98, srv473641108=78, srv817563877=95, srv187400948=47, srv1294899473=17, srv1883853581=48, srv1873200792=46, srv1495479971=26, srv70105925=88, srv1966635448=57, srv180901439=42, srv1101128643=7, srv1945991828=54, srv1493435618=25, srv1621821130=36, srv1545068430=33, srv330260884=69, srv1893078965=50, srv1954864239=56, srv1456715987=24, srv1669983753=39, srv462042567=77, srv569408956=82, srv1067270490=3, srv1098210209=6, srv74445196=90, srv1146163406=8, srv1502286726=27, srv980680761=97, srv581097806=84, srv1524565397=32, srv2044744614=61, srv276542643=68, srv1790804953=41, srv1238452187=15, srv1190084332=11, srv1837150122=44, srv1316361956=21, srv354712443=71, srv493825201=81, srv680229292=87, srv744789257=91, srv1515397877=29, srv16332595=37, srv353333731=70, srv116102831=9, srv2014404518=60, srv40008358=73, srv1608285455=35, srv1513055389=28, srv1926432852=53, srv1012159117=0, srv1053850981=2, srv210945548=63, srv577134704=83, srv1950354868=55, srv2134029312=64, srv1226995976=14, srv1979583766=58, srv909529841=96, srv1277079662=16, srv389167650=72, srv1890196359=49, srv1178156025=10, srv217075551=65, srv801311144=93, srv478187054=79} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-22 09:55:07,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-22 09:55:07,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-22 09:55:07,068 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-22 09:55:07,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-22 09:55:07,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-22 09:55:07,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-22 09:55:07,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-22 09:55:07,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-22 09:55:07,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-22 09:55:07,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-22 09:55:07,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-22 09:55:07,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-22 09:55:07,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-22 09:55:07,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-22 09:55:07,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-22 09:55:07,069 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-22 09:55:07,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-22 09:55:07,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-22 09:55:07,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-22 09:55:07,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-22 09:55:07,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-22 09:55:07,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-22 09:55:07,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-22 09:55:07,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-22 09:55:07,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-22 09:55:07,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-22 09:55:07,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-22 09:55:07,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-22 09:55:07,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-22 09:55:07,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-22 09:55:07,070 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-22 09:55:07,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-22 09:55:07,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-22 09:55:07,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-22 09:55:07,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-22 09:55:07,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-22 09:55:07,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-22 09:55:07,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-22 09:55:07,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-22 09:55:07,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-22 09:55:07,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-22 09:55:07,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-22 09:55:07,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-22 09:55:07,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-22 09:55:07,071 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-22 09:55:07,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-22 09:55:07,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-22 09:55:07,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-22 09:55:07,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-22 09:55:07,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-22 09:55:07,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-22 09:55:07,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-22 09:55:07,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-22 09:55:07,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-22 09:55:07,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-22 09:55:07,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-22 09:55:07,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-22 09:55:07,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-22 09:55:07,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-22 09:55:07,072 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-22 09:55:07,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-22 09:55:07,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-22 09:55:07,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-22 09:55:07,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-22 09:55:07,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-22 09:55:07,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-22 09:55:07,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-22 09:55:07,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-22 09:55:07,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-22 09:55:07,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-22 09:55:07,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-22 09:55:07,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-22 09:55:07,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-22 09:55:07,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-22 09:55:07,073 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-22 09:55:07,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-22 09:55:07,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-22 09:55:07,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-22 09:55:07,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-22 09:55:07,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-22 09:55:07,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-22 09:55:07,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-22 09:55:07,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-22 09:55:07,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-22 09:55:07,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-22 09:55:07,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-22 09:55:07,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-22 09:55:07,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-22 09:55:07,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-22 09:55:07,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-22 09:55:07,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-22 09:55:07,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-22 09:55:07,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-22 09:55:07,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-22 09:55:07,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-22 09:55:07,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-22 09:55:07,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-22 09:55:07,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-22 09:55:07,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-22 09:55:07,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-22 09:55:07,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-22 09:55:07,163 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023168946885067147); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03466368803572313, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023168946885067147); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04816666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-22 09:55:07,163 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-22 09:55:07,163 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.015032159948516546, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.023168946885067147); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03466368803572313, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.023168946885067147); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.011583333333333333); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04816666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-22 09:55:07,318 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 231 ms to try 5000 different iterations.  Found a solution that moves 1000 regions; Going from a computed imbalance of 0.015032159948516546 to a new imbalance of 6.162865803530487E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008125158875920364); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00999438675229417); MoveCostFunction : (multiplier=7.0, imbalance=0.1111111111111111, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008125158875920364); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-22 09:55:07,319 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1926432852:82 , srv1031548325:83 , srv1299197254:83 , srv1515397877:83 , srv152209053:83 , srv1524565397:83 , srv163918372:83 , srv1954864239:83 , srv462042567:83 , srv745405510:83 , srv980680761:83 , srv1090509916:84 , srv1101128643:84 , srv1146163406:84 , srv1190084332:84 , srv119610280:84 , srv1456715987:84 , srv1545068430:84 , srv1669983753:84 , srv180901439:84 , srv1903099598:84 , srv1981134601:84 , srv330260884:84 , srv354712443:84 , srv389167650:84 , srv581097806:84 , srv653357094:84 , srv744789257:84 , srv1067270490:85 , srv1238452187:85 , srv1277079662:85 , srv1307777370:85 , srv1316361956:85 , srv1326708013:85 , srv1495479971:85 , srv1513055389:85 , srv16332595:85 , srv1730094560:85 , srv1790804953:85 , srv1883853581:85 , srv1893078965:85 , srv1908348747:85 , srv2014404518:85 , srv226628313:85 , srv264832822:85 , srv276542643:85 , srv456004216:85 , srv577134704:85 , srv74445196:85 , srv909529841:85 , srv1012159117:86 , srv1067457920:86 , srv1178156025:86 , srv1226995976:86 , srv1294899473:86 , srv129600805:86 , srv1422253031:86 , srv1493435618:86 , srv1519752066:86 , srv1608285455:86 , srv1621821130:86 , srv1837150122:86 , srv1868398866:86 , srv1873200792:86 , srv1950354868:86 , srv1979583766:86 , srv2044744614:86 , srv210945548:86 , srv217075551:86 , srv353333731:86 , srv402746380:86 , srv473641108:86 , srv478187054:86 , srv569408956:86 , srv659304182:86 , srv743741212:86 , srv801311144:86 , srv990591052:86 , srv1098210209:87 , srv116102831:87 , srv122119235:87 , srv1502286726:87 , srv1579877076:87 , srv1824398514:87 , srv187400948:87 , srv1966635448:87 , srv2087627228:87 , srv489612940:87 , srv493825201:87 , srv680229292:87 , srv70105925:87 , srv807739984:87 , srv817563877:87 , srv981900791:87 , srv1053850981:88 , srv1890196359:88 , srv1945991828:88 , srv40008358:88 , srv449436364:88 , srv2134029312:550 }
2024-03-22 09:55:07,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-22 09:55:07,322 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1067457920=4, srv1307777370=20, srv1519752066=30, srv1824398514=43, srv807739984=94, srv1903099598=51, srv745405510=92, srv226628313=66, srv119610280=12, srv152209053=31, srv449436364=75, srv2087627228=62, srv402746380=74, srv456004216=76, srv743741212=89, srv653357094=85, srv1326708013=22, srv990591052=99, srv1730094560=40, srv1981134601=59, srv122119235=13, srv163918372=38, srv659304182=86, srv1422253031=23, srv489612940=80, srv1908348747=52, srv1031548325=1, srv1868398866=45, srv1090509916=5, srv264832822=67, srv1299197254=19, srv1579877076=34, srv129600805=18, srv981900791=98, srv473641108=78, srv817563877=95, srv187400948=47, srv1294899473=17, srv1883853581=48, srv1873200792=46, srv1495479971=26, srv70105925=88, srv1966635448=57, srv180901439=42, srv1101128643=7, srv1945991828=54, srv1493435618=25, srv1621821130=36, srv1545068430=33, srv330260884=69, srv1893078965=50, srv1954864239=56, srv1456715987=24, srv1669983753=39, srv462042567=77, srv569408956=82, srv1067270490=3, srv1098210209=6, srv74445196=90, srv1146163406=8, srv1502286726=27, srv980680761=97, srv581097806=84, srv1524565397=32, srv2044744614=61, srv276542643=68, srv1790804953=41, srv1238452187=15, srv1190084332=11, srv1837150122=44, srv1316361956=21, srv354712443=71, srv493825201=81, srv680229292=87, srv744789257=91, srv1515397877=29, srv16332595=37, srv353333731=70, srv116102831=9, srv2014404518=60, srv40008358=73, srv1608285455=35, srv1513055389=28, srv1926432852=53, srv1012159117=0, srv1053850981=2, srv210945548=63, srv577134704=83, srv1950354868=55, srv2134029312=64, srv1226995976=14, srv1979583766=58, srv909529841=96, srv1277079662=16, srv389167650=72, srv1890196359=49, srv1178156025=10, srv217075551=65, srv801311144=93, srv478187054=79} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-22 09:55:07,331 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-22 09:55:07,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-22 09:55:07,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-22 09:55:07,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-22 09:55:07,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-22 09:55:07,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-22 09:55:07,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-22 09:55:07,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-22 09:55:07,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-22 09:55:07,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-22 09:55:07,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-22 09:55:07,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-22 09:55:07,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-22 09:55:07,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-22 09:55:07,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-22 09:55:07,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-22 09:55:07,332 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-22 09:55:07,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-22 09:55:07,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-22 09:55:07,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-22 09:55:07,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-22 09:55:07,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-22 09:55:07,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-22 09:55:07,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-22 09:55:07,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-22 09:55:07,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-22 09:55:07,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-22 09:55:07,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-22 09:55:07,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-22 09:55:07,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-22 09:55:07,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-22 09:55:07,333 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-22 09:55:07,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-22 09:55:07,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-22 09:55:07,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-22 09:55:07,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-22 09:55:07,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-22 09:55:07,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-22 09:55:07,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-22 09:55:07,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-22 09:55:07,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-22 09:55:07,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-22 09:55:07,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-22 09:55:07,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-22 09:55:07,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-22 09:55:07,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-22 09:55:07,334 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-22 09:55:07,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-22 09:55:07,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-22 09:55:07,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-22 09:55:07,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-22 09:55:07,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-22 09:55:07,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-22 09:55:07,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-22 09:55:07,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-22 09:55:07,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-22 09:55:07,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-22 09:55:07,335 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-22 09:55:07,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-22 09:55:07,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-22 09:55:07,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-22 09:55:07,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-22 09:55:07,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-22 09:55:07,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-22 09:55:07,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-22 09:55:07,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-22 09:55:07,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-22 09:55:07,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-22 09:55:07,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-22 09:55:07,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-22 09:55:07,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-22 09:55:07,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-22 09:55:07,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-22 09:55:07,336 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-22 09:55:07,337 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-22 09:55:07,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-22 09:55:07,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-22 09:55:07,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-22 09:55:07,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-22 09:55:07,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-22 09:55:07,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-22 09:55:07,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-22 09:55:07,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-22 09:55:07,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-22 09:55:07,338 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-22 09:55:07,418 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008125158875920364); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00999438675229417); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008125158875920364); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-22 09:55:07,418 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-22 09:55:07,419 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=6.092844306931031E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.008125158875920364); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.00999438675229417); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.008125158875920364); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=8.333333333333333E-5); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.005); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-22 09:55:07,634 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 288 ms to try 5000 different iterations.  Found a solution that moves 314 regions; Going from a computed imbalance of 6.092844306931031E-4 to a new imbalance of 3.548813728282555E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.034888888888888886, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-22 09:55:07,635 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1513055389:83 , srv1868398866:83 , srv1890196359:83 , srv40008358:83 , srv449436364:83 , srv1053850981:84 , srv1067270490:84 , srv122119235:84 , srv1226995976:84 , srv1277079662:84 , srv1294899473:84 , srv129600805:84 , srv1307777370:84 , srv1326708013:84 , srv1456715987:84 , srv1495479971:84 , srv1502286726:84 , srv1515397877:84 , srv1837150122:84 , srv1873200792:84 , srv187400948:84 , srv1893078965:84 , srv1945991828:84 , srv2044744614:84 , srv2087627228:84 , srv226628313:84 , srv276542643:84 , srv402746380:84 , srv456004216:84 , srv478187054:84 , srv489612940:84 , srv493825201:84 , srv577134704:84 , srv581097806:84 , srv659304182:84 , srv70105925:84 , srv74445196:84 , srv745405510:84 , srv817563877:84 , srv990591052:84 , srv1031548325:85 , srv1067457920:85 , srv1101128643:85 , srv1178156025:85 , srv1190084332:85 , srv1299197254:85 , srv1316361956:85 , srv1422253031:85 , srv1493435618:85 , srv1519752066:85 , srv1524565397:85 , srv1579877076:85 , srv1608285455:85 , srv163918372:85 , srv1669983753:85 , srv1790804953:85 , srv180901439:85 , srv1824398514:85 , srv1883853581:85 , srv1903099598:85 , srv1908348747:85 , srv1926432852:85 , srv1950354868:85 , srv1954864239:85 , srv1966635448:85 , srv1979583766:85 , srv1981134601:85 , srv2014404518:85 , srv210945548:85 , srv217075551:85 , srv264832822:85 , srv330260884:85 , srv353333731:85 , srv389167650:85 , srv569408956:85 , srv653357094:85 , srv680229292:85 , srv743741212:85 , srv807739984:85 , srv909529841:85 , srv980680761:85 , srv981900791:85 , srv1012159117:86 , srv1090509916:86 , srv1098210209:86 , srv1146163406:86 , srv116102831:86 , srv119610280:86 , srv1238452187:86 , srv152209053:86 , srv1545068430:86 , srv1621821130:86 , srv16332595:86 , srv1730094560:86 , srv354712443:86 , srv462042567:86 , srv744789257:86 , srv801311144:86 , srv473641108:87 , srv2134029312:612 }
2024-03-22 09:55:07,639 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-22 09:55:07,640 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1067457920=4, srv1307777370=20, srv1519752066=30, srv1824398514=43, srv807739984=94, srv1903099598=51, srv745405510=92, srv226628313=66, srv119610280=12, srv152209053=31, srv449436364=75, srv2087627228=62, srv402746380=74, srv456004216=76, srv743741212=89, srv653357094=85, srv1326708013=22, srv990591052=99, srv1730094560=40, srv1981134601=59, srv122119235=13, srv163918372=38, srv659304182=86, srv1422253031=23, srv489612940=80, srv1908348747=52, srv1031548325=1, srv1868398866=45, srv1090509916=5, srv264832822=67, srv1299197254=19, srv1579877076=34, srv129600805=18, srv981900791=98, srv473641108=78, srv817563877=95, srv187400948=47, srv1294899473=17, srv1883853581=48, srv1873200792=46, srv1495479971=26, srv70105925=88, srv1966635448=57, srv180901439=42, srv1101128643=7, srv1945991828=54, srv1493435618=25, srv1621821130=36, srv1545068430=33, srv330260884=69, srv1893078965=50, srv1954864239=56, srv1456715987=24, srv1669983753=39, srv462042567=77, srv569408956=82, srv1067270490=3, srv1098210209=6, srv74445196=90, srv1146163406=8, srv1502286726=27, srv980680761=97, srv581097806=84, srv1524565397=32, srv2044744614=61, srv276542643=68, srv1790804953=41, srv1238452187=15, srv1190084332=11, srv1837150122=44, srv1316361956=21, srv354712443=71, srv493825201=81, srv680229292=87, srv744789257=91, srv1515397877=29, srv16332595=37, srv353333731=70, srv116102831=9, srv2014404518=60, srv40008358=73, srv1608285455=35, srv1513055389=28, srv1926432852=53, srv1012159117=0, srv1053850981=2, srv210945548=63, srv577134704=83, srv1950354868=55, srv2134029312=64, srv1226995976=14, srv1979583766=58, srv909529841=96, srv1277079662=16, srv389167650=72, srv1890196359=49, srv1178156025=10, srv217075551=65, srv801311144=93, srv478187054=79} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-22 09:55:07,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-22 09:55:07,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-22 09:55:07,654 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-22 09:55:07,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-22 09:55:07,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-22 09:55:07,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-22 09:55:07,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-22 09:55:07,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-22 09:55:07,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-22 09:55:07,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-22 09:55:07,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-22 09:55:07,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-22 09:55:07,655 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-22 09:55:07,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-22 09:55:07,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-22 09:55:07,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-22 09:55:07,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-22 09:55:07,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-22 09:55:07,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-22 09:55:07,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-22 09:55:07,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-22 09:55:07,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-22 09:55:07,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-22 09:55:07,656 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-22 09:55:07,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-22 09:55:07,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-22 09:55:07,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-22 09:55:07,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-22 09:55:07,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-22 09:55:07,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-22 09:55:07,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-22 09:55:07,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-22 09:55:07,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-22 09:55:07,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-22 09:55:07,657 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-22 09:55:07,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-22 09:55:07,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-22 09:55:07,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-22 09:55:07,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-22 09:55:07,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-22 09:55:07,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-22 09:55:07,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-22 09:55:07,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-22 09:55:07,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-22 09:55:07,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-22 09:55:07,658 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-22 09:55:07,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-22 09:55:07,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-22 09:55:07,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-22 09:55:07,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-22 09:55:07,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-22 09:55:07,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-22 09:55:07,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-22 09:55:07,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-22 09:55:07,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-22 09:55:07,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-22 09:55:07,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-22 09:55:07,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-22 09:55:07,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-22 09:55:07,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-22 09:55:07,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-22 09:55:07,659 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-22 09:55:07,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-22 09:55:07,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-22 09:55:07,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-22 09:55:07,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-22 09:55:07,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-22 09:55:07,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-22 09:55:07,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-22 09:55:07,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-22 09:55:07,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-22 09:55:07,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-22 09:55:07,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-22 09:55:07,660 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-22 09:55:07,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-22 09:55:07,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-22 09:55:07,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-22 09:55:07,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-22 09:55:07,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-22 09:55:07,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-22 09:55:07,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-22 09:55:07,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-22 09:55:07,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-22 09:55:07,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-22 09:55:07,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-22 09:55:07,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-22 09:55:07,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-22 09:55:07,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-22 09:55:07,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-22 09:55:07,661 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-22 09:55:07,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-22 09:55:07,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-22 09:55:07,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-22 09:55:07,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-22 09:55:07,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-22 09:55:07,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-22 09:55:07,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-22 09:55:07,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-22 09:55:07,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-22 09:55:07,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-22 09:55:07,662 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-22 09:55:07,741 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-22 09:55:07,741 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2024-03-22 09:55:07,741 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=3.328946228960263E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=5.23782800878924E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0018349396085439343); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=5.23782800878924E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=2.5E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2024-03-22 09:55:07,891 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 222 ms to try 5000 different iterations.  Found a solution that moves 29 regions; Going from a computed imbalance of 3.328946228960263E-5 to a new imbalance of 6.601048012408121E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0032222222222222222); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-22 09:55:07,891 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv569408956:82 , srv1090509916:83 , srv152209053:83 , srv163918372:83 , srv353333731:83 , srv354712443:83 , srv473641108:83 , srv743741212:83 , srv801311144:83 , srv1012159117:84 , srv1031548325:84 , srv1053850981:84 , srv1067270490:84 , srv1067457920:84 , srv1098210209:84 , srv1101128643:84 , srv1146163406:84 , srv116102831:84 , srv1178156025:84 , srv1190084332:84 , srv119610280:84 , srv122119235:84 , srv1226995976:84 , srv1238452187:84 , srv1294899473:84 , srv129600805:84 , srv1299197254:84 , srv1316361956:84 , srv1326708013:84 , srv1422253031:84 , srv1456715987:84 , srv1493435618:84 , srv1495479971:84 , srv1502286726:84 , srv1515397877:84 , srv1519752066:84 , srv1524565397:84 , srv1545068430:84 , srv1579877076:84 , srv1608285455:84 , srv1621821130:84 , srv16332595:84 , srv1669983753:84 , srv1730094560:84 , srv1790804953:84 , srv180901439:84 , srv1824398514:84 , srv1837150122:84 , srv1868398866:84 , srv1873200792:84 , srv1890196359:84 , srv1893078965:84 , srv1908348747:84 , srv1926432852:84 , srv1945991828:84 , srv1950354868:84 , srv1954864239:84 , srv1966635448:84 , srv1979583766:84 , srv1981134601:84 , srv2014404518:84 , srv2044744614:84 , srv2087627228:84 , srv210945548:84 , srv217075551:84 , srv226628313:84 , srv264832822:84 , srv276542643:84 , srv330260884:84 , srv389167650:84 , srv402746380:84 , srv449436364:84 , srv462042567:84 , srv478187054:84 , srv489612940:84 , srv493825201:84 , srv577134704:84 , srv581097806:84 , srv653357094:84 , srv659304182:84 , srv680229292:84 , srv74445196:84 , srv744789257:84 , srv807739984:84 , srv817563877:84 , srv909529841:84 , srv980680761:84 , srv981900791:84 , srv990591052:84 , srv1277079662:85 , srv1307777370:85 , srv1513055389:85 , srv187400948:85 , srv1883853581:85 , srv1903099598:85 , srv40008358:85 , srv456004216:85 , srv70105925:85 , srv745405510:85 , srv2134029312:684 }
2024-03-22 09:55:07,894 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-22 09:55:07,895 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1067457920=4, srv1307777370=20, srv1519752066=30, srv1824398514=43, srv807739984=94, srv1903099598=51, srv745405510=92, srv226628313=66, srv119610280=12, srv152209053=31, srv449436364=75, srv2087627228=62, srv402746380=74, srv456004216=76, srv743741212=89, srv653357094=85, srv1326708013=22, srv990591052=99, srv1730094560=40, srv1981134601=59, srv122119235=13, srv163918372=38, srv659304182=86, srv1422253031=23, srv489612940=80, srv1908348747=52, srv1031548325=1, srv1868398866=45, srv1090509916=5, srv264832822=67, srv1299197254=19, srv1579877076=34, srv129600805=18, srv981900791=98, srv473641108=78, srv817563877=95, srv187400948=47, srv1294899473=17, srv1883853581=48, srv1873200792=46, srv1495479971=26, srv70105925=88, srv1966635448=57, srv180901439=42, srv1101128643=7, srv1945991828=54, srv1493435618=25, srv1621821130=36, srv1545068430=33, srv330260884=69, srv1893078965=50, srv1954864239=56, srv1456715987=24, srv1669983753=39, srv462042567=77, srv569408956=82, srv1067270490=3, srv1098210209=6, srv74445196=90, srv1146163406=8, srv1502286726=27, srv980680761=97, srv581097806=84, srv1524565397=32, srv2044744614=61, srv276542643=68, srv1790804953=41, srv1238452187=15, srv1190084332=11, srv1837150122=44, srv1316361956=21, srv354712443=71, srv493825201=81, srv680229292=87, srv744789257=91, srv1515397877=29, srv16332595=37, srv353333731=70, srv116102831=9, srv2014404518=60, srv40008358=73, srv1608285455=35, srv1513055389=28, srv1926432852=53, srv1012159117=0, srv1053850981=2, srv210945548=63, srv577134704=83, srv1950354868=55, srv2134029312=64, srv1226995976=14, srv1979583766=58, srv909529841=96, srv1277079662=16, srv389167650=72, srv1890196359=49, srv1178156025=10, srv217075551=65, srv801311144=93, srv478187054=79} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-22 09:55:07,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-22 09:55:07,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-22 09:55:07,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-22 09:55:07,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-22 09:55:07,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-22 09:55:07,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-22 09:55:07,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-22 09:55:07,903 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-22 09:55:07,904 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2024-03-22 09:55:07,905 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2024-03-22 09:55:07,906 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2024-03-22 09:55:07,907 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2024-03-22 09:55:07,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2024-03-22 09:55:07,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2024-03-22 09:55:07,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2024-03-22 09:55:07,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2024-03-22 09:55:07,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2024-03-22 09:55:07,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2024-03-22 09:55:07,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2024-03-22 09:55:07,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2024-03-22 09:55:07,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2024-03-22 09:55:07,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2024-03-22 09:55:07,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2024-03-22 09:55:07,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2024-03-22 09:55:07,908 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2024-03-22 09:55:07,984 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.397985672269699E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014213381090374027); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-22 09:55:07,985 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv569408956:82 , srv1090509916:83 , srv152209053:83 , srv163918372:83 , srv353333731:83 , srv354712443:83 , srv473641108:83 , srv743741212:83 , srv801311144:83 , srv1012159117:84 , srv1031548325:84 , srv1053850981:84 , srv1067270490:84 , srv1067457920:84 , srv1098210209:84 , srv1101128643:84 , srv1146163406:84 , srv116102831:84 , srv1178156025:84 , srv1190084332:84 , srv119610280:84 , srv122119235:84 , srv1226995976:84 , srv1238452187:84 , srv1294899473:84 , srv129600805:84 , srv1299197254:84 , srv1316361956:84 , srv1326708013:84 , srv1422253031:84 , srv1456715987:84 , srv1493435618:84 , srv1495479971:84 , srv1502286726:84 , srv1515397877:84 , srv1519752066:84 , srv1524565397:84 , srv1545068430:84 , srv1579877076:84 , srv1608285455:84 , srv1621821130:84 , srv16332595:84 , srv1669983753:84 , srv1730094560:84 , srv1790804953:84 , srv180901439:84 , srv1824398514:84 , srv1837150122:84 , srv1868398866:84 , srv1873200792:84 , srv1890196359:84 , srv1893078965:84 , srv1908348747:84 , srv1926432852:84 , srv1945991828:84 , srv1950354868:84 , srv1954864239:84 , srv1966635448:84 , srv1979583766:84 , srv1981134601:84 , srv2014404518:84 , srv2044744614:84 , srv2087627228:84 , srv210945548:84 , srv217075551:84 , srv226628313:84 , srv264832822:84 , srv276542643:84 , srv330260884:84 , srv389167650:84 , srv402746380:84 , srv449436364:84 , srv462042567:84 , srv478187054:84 , srv489612940:84 , srv493825201:84 , srv577134704:84 , srv581097806:84 , srv653357094:84 , srv659304182:84 , srv680229292:84 , srv74445196:84 , srv744789257:84 , srv807739984:84 , srv817563877:84 , srv909529841:84 , srv980680761:84 , srv981900791:84 , srv990591052:84 , srv1277079662:85 , srv1307777370:85 , srv1513055389:85 , srv187400948:85 , srv1883853581:85 , srv1903099598:85 , srv40008358:85 , srv456004216:85 , srv70105925:85 , srv745405510:85 , srv2134029312:684 }
2024-03-22 09:55:08,007 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=12 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=149 (was 149), ProcessCount=177 (was 177), AvailableMemoryMB=6412 (was 6904)
2024-03-22 09:55:08,015 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=12, OpenFileDescriptor=290, MaxFileDescriptor=60000, SystemLoadAverage=149, ProcessCount=177, AvailableMemoryMB=6412
2024-03-22 09:55:08,015 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2024-03-22 09:55:08,016 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2024-03-22 09:55:08,025 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1174292672:84 , srv1210927660:84 , srv1217607007:84 , srv1285805692:84 , srv1410522856:84 , srv1489906972:84 , srv1636183597:84 , srv1655111139:84 , srv1748484365:84 , srv192314816:84 , srv1925924592:84 , srv2024409673:84 , srv2087906244:84 , srv2135754998:84 , srv250926109:84 , srv340480819:84 , srv341111673:84 , srv353409201:84 , srv455551058:84 , srv519567340:84 , srv593616712:84 , srv598368125:84 , srv614516421:84 , srv618946352:84 , srv814746373:84 , srv848268518:84 , srv864215638:84 , srv902448326:84 , srv997367357:84 , srv1107805033:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2024-03-22 09:55:08,026 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-22 09:55:08,027 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1285805692=4, srv353409201=18, srv593616712=21, srv1748484365=9, srv1489906972=6, srv2087906244=13, srv997367357=29, srv614516421=23, srv1107805033=0, srv250926109=15, srv1210927660=2, srv598368125=22, srv1410522856=5, srv1217607007=3, srv814746373=25, srv1174292672=1, srv2024409673=12, srv618946352=24, srv341111673=17, srv1655111139=8, srv2135754998=14, srv1636183597=7, srv519567340=20, srv1925924592=11, srv848268518=26, srv864215638=27, srv340480819=16, srv902448326=28, srv192314816=10, srv455551058=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-22 09:55:08,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-22 09:55:08,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-22 09:55:08,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-22 09:55:08,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-22 09:55:08,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-22 09:55:08,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-22 09:55:08,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-22 09:55:08,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-22 09:55:08,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-22 09:55:08,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-22 09:55:08,036 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-22 09:55:08,037 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-22 09:55:08,050 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1215985701753441, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-22 09:55:08,050 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912576196388344, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1215985701753441, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2024-03-22 09:57:08,040 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 9255179 different iterations.  Found a solution that moves 1829 regions; Going from a computed imbalance of 0.9912576196388344 to a new imbalance of 4.26897724268017E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6774074074074075, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-22 09:57:08,046 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1107805033:90 , srv1174292672:90 , srv1210927660:90 , srv1217607007:90 , srv1285805692:90 , srv1410522856:90 , srv1489906972:90 , srv1636183597:90 , srv1655111139:90 , srv1748484365:90 , srv192314816:90 , srv1925924592:90 , srv2024409673:90 , srv2087906244:90 , srv2135754998:90 , srv250926109:90 , srv340480819:90 , srv341111673:90 , srv353409201:90 , srv455551058:90 , srv519567340:90 , srv593616712:90 , srv598368125:90 , srv614516421:90 , srv618946352:90 , srv814746373:90 , srv848268518:90 , srv864215638:90 , srv902448326:90 , srv997367357:90 }
2024-03-22 09:57:08,047 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2024-03-22 09:57:08,048 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv1285805692=4, srv353409201=18, srv593616712=21, srv1748484365=9, srv1489906972=6, srv2087906244=13, srv997367357=29, srv614516421=23, srv1107805033=0, srv250926109=15, srv1210927660=2, srv598368125=22, srv1410522856=5, srv1217607007=3, srv814746373=25, srv1174292672=1, srv2024409673=12, srv618946352=24, srv341111673=17, srv1655111139=8, srv2135754998=14, srv1636183597=7, srv519567340=20, srv1925924592=11, srv848268518=26, srv864215638=27, srv340480819=16, srv902448326=28, srv192314816=10, srv455551058=19} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2024-03-22 09:57:08,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2024-03-22 09:57:08,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2024-03-22 09:57:08,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2024-03-22 09:57:08,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2024-03-22 09:57:08,052 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2024-03-22 09:57:08,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2024-03-22 09:57:08,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2024-03-22 09:57:08,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2024-03-22 09:57:08,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2024-03-22 09:57:08,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2024-03-22 09:57:08,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2024-03-22 09:57:08,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2024-03-22 09:57:08,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2024-03-22 09:57:08,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2024-03-22 09:57:08,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2024-03-22 09:57:08,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2024-03-22 09:57:08,053 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2024-03-22 09:57:08,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2024-03-22 09:57:08,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2024-03-22 09:57:08,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2024-03-22 09:57:08,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2024-03-22 09:57:08,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2024-03-22 09:57:08,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2024-03-22 09:57:08,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2024-03-22 09:57:08,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2024-03-22 09:57:08,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2024-03-22 09:57:08,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2024-03-22 09:57:08,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2024-03-22 09:57:08,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2024-03-22 09:57:08,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2024-03-22 09:57:08,054 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2024-03-22 09:57:08,065 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2024-03-22 09:57:08,066 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1107805033:90 , srv1174292672:90 , srv1210927660:90 , srv1217607007:90 , srv1285805692:90 , srv1410522856:90 , srv1489906972:90 , srv1636183597:90 , srv1655111139:90 , srv1748484365:90 , srv192314816:90 , srv1925924592:90 , srv2024409673:90 , srv2087906244:90 , srv2135754998:90 , srv250926109:90 , srv340480819:90 , srv341111673:90 , srv353409201:90 , srv455551058:90 , srv519567340:90 , srv593616712:90 , srv598368125:90 , srv614516421:90 , srv618946352:90 , srv814746373:90 , srv848268518:90 , srv864215638:90 , srv902448326:90 , srv997367357:90 }
2024-03-22 09:57:08,078 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=11 (was 12), OpenFileDescriptor=290 (was 290), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=231 (was 149) - SystemLoadAverage LEAK? -, ProcessCount=176 (was 177), AvailableMemoryMB=7484 (was 6412) - AvailableMemoryMB LEAK? -