2023-03-16 20:53:59,623 INFO  [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks timeout: 13 mins
2023-03-16 20:53:59,942 WARN  [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
2023-03-16 20:54:00,049 INFO  [Time-limited test] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2023-03-16 20:54:00,108 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-16 20:54:00,116 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-16 20:54:00,132 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=315, ProcessCount=169, AvailableMemoryMB=5571
2023-03-16 20:54:00,134 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-16 20:54:00,135 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=10000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc.
2023-03-16 20:54:00,252 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1011581941:84 , srv1036894495:84 , srv1040259946:84 , srv1051308801:84 , srv1053668713:84 , srv1081728014:84 , srv1087817474:84 , srv1104465367:84 , srv1181798119:84 , srv1182258107:84 , srv1191706008:84 , srv1216253621:84 , srv1260771203:84 , srv1318718344:84 , srv133913198:84 , srv134326997:84 , srv1374645635:84 , srv1396812949:84 , srv1401795885:84 , srv1403002210:84 , srv1408008551:84 , srv1448282540:84 , srv1483579262:84 , srv149646235:84 , srv1500361580:84 , srv1512139203:84 , srv1573942930:84 , srv1611655391:84 , srv1629122668:84 , srv1637792304:84 , srv1718254444:84 , srv178437805:84 , srv1785869427:84 , srv1788957987:84 , srv1820334161:84 , srv1820366166:84 , srv1831766628:84 , srv1837108774:84 , srv1898409639:84 , srv1918079185:84 , srv1925418933:84 , srv1942636744:84 , srv194765176:84 , srv1958144443:84 , srv1981199181:84 , srv2023026317:84 , srv2060352623:84 , srv2069467738:84 , srv2079400271:84 , srv2107071244:84 , srv2133405805:84 , srv216972212:84 , srv228251024:84 , srv236415718:84 , srv244026697:84 , srv282377255:84 , srv297574165:84 , srv306712946:84 , srv325833901:84 , srv402621384:84 , srv427085076:84 , srv441171474:84 , srv458885464:84 , srv47534880:84 , srv485241821:84 , srv486891914:84 , srv524318482:84 , srv529032350:84 , srv535740985:84 , srv535948106:84 , srv54630959:84 , srv54841523:84 , srv570214459:84 , srv572871616:84 , srv602563603:84 , srv613601579:84 , srv640695358:84 , srv658294223:84 , srv690378883:84 , srv72901648:84 , srv733271346:84 , srv743974627:84 , srv765700448:84 , srv775015900:84 , srv779505717:84 , srv819151041:84 , srv838706054:84 , srv840339351:84 , srv855882867:84 , srv862307642:84 , srv877316175:84 , srv88129284:84 , srv887595064:84 , srv894044332:84 , srv895627113:84 , srv918323524:84 , srv934226878:84 , srv939795244:84 , srv985353476:84 , srv1096490251:684 } [srvr=100 rgns=9000 avg=90.0 max=90 min=90]
2023-03-16 20:54:00,261 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-16 20:54:00,269 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv458885464=63, srv2079400271=49, srv1820334161=35, srv524318482=67, srv1981199181=45, srv306712946=58, srv149646235=24, srv1820366166=36, srv216972212=52, srv877316175=91, srv178437805=32, srv918323524=96, srv894044332=94, srv1396812949=18, srv1260771203=13, srv1408008551=21, srv690378883=79, srv535948106=70, srv895627113=95, srv1011581941=0, srv855882867=89, srv733271346=81, srv939795244=98, srv1096490251=7, srv819151041=86, srv2133405805=51, srv1483579262=23, srv613601579=76, srv1081728014=5, srv570214459=73, srv1500361580=25, srv1182258107=10, srv1087817474=6, srv297574165=57, srv775015900=84, srv236415718=54, srv54841523=72, srv1925418933=41, srv244026697=55, srv1191706008=11, srv1958144443=44, srv840339351=88, srv486891914=66, srv54630959=71, srv1051308801=3, srv1837108774=38, srv402621384=60, srv1403002210=20, srv1053668713=4, srv985353476=99, srv765700448=83, srv88129284=92, srv1512139203=26, srv427085076=61, srv282377255=56, srv1785869427=33, srv1629122668=29, srv1898409639=39, srv2069467738=48, srv134326997=16, srv1573942930=27, srv194765176=43, srv1374645635=17, srv1181798119=9, srv640695358=77, srv2107071244=50, srv934226878=97, srv1216253621=12, srv441171474=62, srv838706054=87, srv1040259946=2, srv2023026317=46, srv325833901=59, srv72901648=80, srv887595064=93, srv1611655391=28, srv602563603=75, srv1918079185=40, srv228251024=53, srv1448282540=22, srv47534880=64, srv2060352623=47, srv485241821=65, srv1788957987=34, srv535740985=69, srv1637792304=30, srv779505717=85, srv1831766628=37, srv862307642=90, srv1318718344=14, srv1104465367=8, srv1036894495=1, srv1401795885=19, srv572871616=74, srv133913198=15, srv529032350=68, srv1942636744=42, srv743974627=82, srv1718254444=31, srv658294223=78} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-16 20:54:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-16 20:54:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-16 20:54:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-16 20:54:00,357 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-16 20:54:00,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-16 20:54:00,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-16 20:54:00,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-16 20:54:00,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-16 20:54:00,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-16 20:54:00,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-16 20:54:00,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-16 20:54:00,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-16 20:54:00,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-16 20:54:00,358 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-16 20:54:00,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-16 20:54:00,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-16 20:54:00,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-16 20:54:00,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-16 20:54:00,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-16 20:54:00,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-16 20:54:00,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-16 20:54:00,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-16 20:54:00,359 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-16 20:54:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-16 20:54:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-16 20:54:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-16 20:54:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-16 20:54:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-16 20:54:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-16 20:54:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-16 20:54:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-16 20:54:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-16 20:54:00,360 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-16 20:54:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-16 20:54:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-16 20:54:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-16 20:54:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-16 20:54:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-16 20:54:00,361 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-16 20:54:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-16 20:54:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-16 20:54:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-16 20:54:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-16 20:54:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-16 20:54:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-16 20:54:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-16 20:54:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-16 20:54:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-16 20:54:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-16 20:54:00,362 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-16 20:54:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-16 20:54:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-16 20:54:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-16 20:54:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-16 20:54:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-16 20:54:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-16 20:54:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-16 20:54:00,363 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-16 20:54:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-16 20:54:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-16 20:54:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-16 20:54:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-16 20:54:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-16 20:54:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-16 20:54:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-16 20:54:00,364 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-16 20:54:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-16 20:54:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-16 20:54:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-16 20:54:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-16 20:54:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-16 20:54:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-16 20:54:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-16 20:54:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-16 20:54:00,365 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-16 20:54:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-16 20:54:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-16 20:54:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-16 20:54:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-16 20:54:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-16 20:54:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-16 20:54:00,366 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-16 20:54:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-16 20:54:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-16 20:54:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-16 20:54:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-16 20:54:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-16 20:54:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-16 20:54:00,367 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-16 20:54:00,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-16 20:54:00,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-16 20:54:00,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-16 20:54:00,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-16 20:54:00,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-16 20:54:00,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-16 20:54:00,368 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-16 20:54:00,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-16 20:54:00,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-16 20:54:00,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-16 20:54:00,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-16 20:54:00,369 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-16 20:54:00,534 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 20:54:00,534 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-16 20:54:00,534 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9909252140407105, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.06666666666666667, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-16 20:54:01,205 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 811 ms to try 5000 different iterations.  Found a solution that moves 4661 regions; Going from a computed imbalance of 0.9909252140407105 to a new imbalance of 0.18091135953785578. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04032323569698305, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051452526799265944, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.5178888888888888, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04032323569698305, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17566666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 20:54:01,226 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv894044332:80 , srv1104465367:81 , srv1483579262:81 , srv178437805:81 , srv733271346:81 , srv1318718344:82 , srv149646235:82 , srv236415718:82 , srv524318482:82 , srv840339351:82 , srv1087817474:83 , srv1831766628:83 , srv1918079185:83 , srv572871616:83 , srv613601579:83 , srv877316175:83 , srv1011581941:84 , srv1181798119:84 , srv1182258107:84 , srv1216253621:84 , srv134326997:84 , srv1611655391:84 , srv282377255:84 , srv427085076:84 , srv529032350:84 , srv72901648:84 , srv743974627:84 , srv775015900:84 , srv855882867:84 , srv1396812949:85 , srv1403002210:85 , srv1629122668:85 , srv1785869427:85 , srv1820334161:85 , srv1837108774:85 , srv1981199181:85 , srv2079400271:85 , srv306712946:85 , srv441171474:85 , srv658294223:85 , srv779505717:85 , srv88129284:85 , srv1051308801:86 , srv1053668713:86 , srv1260771203:86 , srv1573942930:86 , srv325833901:86 , srv486891914:86 , srv570214459:86 , srv602563603:86 , srv765700448:86 , srv1036894495:87 , srv1081728014:87 , srv133913198:87 , srv1408008551:87 , srv1448282540:87 , srv1500361580:87 , srv1718254444:87 , srv1820366166:87 , srv1958144443:87 , srv297574165:87 , srv535740985:87 , srv535948106:87 , srv819151041:87 , srv838706054:87 , srv1040259946:88 , srv1512139203:88 , srv194765176:88 , srv2023026317:88 , srv2069467738:88 , srv2107071244:88 , srv228251024:88 , srv244026697:88 , srv54841523:88 , srv640695358:88 , srv887595064:88 , srv1637792304:89 , srv2060352623:89 , srv216972212:89 , srv402621384:89 , srv485241821:89 , srv54630959:89 , srv690378883:89 , srv862307642:89 , srv918323524:89 , srv934226878:89 , srv1374645635:90 , srv2133405805:90 , srv1401795885:91 , srv1788957987:91 , srv1898409639:91 , srv47534880:91 , srv939795244:91 , srv1191706008:92 , srv1925418933:92 , srv458885464:92 , srv895627113:93 , srv1942636744:94 , srv985353476:95 , srv1096490251:448 }
2023-03-16 20:54:01,230 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-16 20:54:01,231 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv458885464=63, srv2079400271=49, srv1820334161=35, srv524318482=67, srv1981199181=45, srv306712946=58, srv149646235=24, srv1820366166=36, srv216972212=52, srv877316175=91, srv178437805=32, srv918323524=96, srv894044332=94, srv1396812949=18, srv1260771203=13, srv1408008551=21, srv690378883=79, srv535948106=70, srv895627113=95, srv1011581941=0, srv855882867=89, srv733271346=81, srv939795244=98, srv1096490251=7, srv819151041=86, srv2133405805=51, srv1483579262=23, srv613601579=76, srv1081728014=5, srv570214459=73, srv1500361580=25, srv1182258107=10, srv1087817474=6, srv297574165=57, srv775015900=84, srv236415718=54, srv54841523=72, srv1925418933=41, srv244026697=55, srv1191706008=11, srv1958144443=44, srv840339351=88, srv486891914=66, srv54630959=71, srv1051308801=3, srv1837108774=38, srv402621384=60, srv1403002210=20, srv1053668713=4, srv985353476=99, srv765700448=83, srv88129284=92, srv1512139203=26, srv427085076=61, srv282377255=56, srv1785869427=33, srv1629122668=29, srv1898409639=39, srv2069467738=48, srv134326997=16, srv1573942930=27, srv194765176=43, srv1374645635=17, srv1181798119=9, srv640695358=77, srv2107071244=50, srv934226878=97, srv1216253621=12, srv441171474=62, srv838706054=87, srv1040259946=2, srv2023026317=46, srv325833901=59, srv72901648=80, srv887595064=93, srv1611655391=28, srv602563603=75, srv1918079185=40, srv228251024=53, srv1448282540=22, srv47534880=64, srv2060352623=47, srv485241821=65, srv1788957987=34, srv535740985=69, srv1637792304=30, srv779505717=85, srv1831766628=37, srv862307642=90, srv1318718344=14, srv1104465367=8, srv1036894495=1, srv1401795885=19, srv572871616=74, srv133913198=15, srv529032350=68, srv1942636744=42, srv743974627=82, srv1718254444=31, srv658294223=78} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-16 20:54:01,245 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-16 20:54:01,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-16 20:54:01,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-16 20:54:01,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-16 20:54:01,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-16 20:54:01,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-16 20:54:01,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-16 20:54:01,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-16 20:54:01,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-16 20:54:01,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-16 20:54:01,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-16 20:54:01,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-16 20:54:01,246 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-16 20:54:01,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-16 20:54:01,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-16 20:54:01,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-16 20:54:01,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-16 20:54:01,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-16 20:54:01,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-16 20:54:01,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-16 20:54:01,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-16 20:54:01,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-16 20:54:01,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-16 20:54:01,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-16 20:54:01,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-16 20:54:01,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-16 20:54:01,247 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-16 20:54:01,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-16 20:54:01,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-16 20:54:01,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-16 20:54:01,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-16 20:54:01,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-16 20:54:01,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-16 20:54:01,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-16 20:54:01,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-16 20:54:01,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-16 20:54:01,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-16 20:54:01,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-16 20:54:01,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-16 20:54:01,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-16 20:54:01,248 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-16 20:54:01,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-16 20:54:01,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-16 20:54:01,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-16 20:54:01,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-16 20:54:01,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-16 20:54:01,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-16 20:54:01,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-16 20:54:01,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-16 20:54:01,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-16 20:54:01,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-16 20:54:01,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-16 20:54:01,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-16 20:54:01,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-16 20:54:01,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-16 20:54:01,249 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-16 20:54:01,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-16 20:54:01,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-16 20:54:01,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-16 20:54:01,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-16 20:54:01,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-16 20:54:01,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-16 20:54:01,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-16 20:54:01,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-16 20:54:01,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-16 20:54:01,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-16 20:54:01,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-16 20:54:01,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-16 20:54:01,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-16 20:54:01,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-16 20:54:01,250 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-16 20:54:01,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-16 20:54:01,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-16 20:54:01,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-16 20:54:01,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-16 20:54:01,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-16 20:54:01,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-16 20:54:01,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-16 20:54:01,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-16 20:54:01,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-16 20:54:01,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-16 20:54:01,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-16 20:54:01,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-16 20:54:01,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-16 20:54:01,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-16 20:54:01,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-16 20:54:01,251 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-16 20:54:01,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-16 20:54:01,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-16 20:54:01,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-16 20:54:01,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-16 20:54:01,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-16 20:54:01,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-16 20:54:01,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-16 20:54:01,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-16 20:54:01,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-16 20:54:01,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-16 20:54:01,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-16 20:54:01,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-16 20:54:01,252 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-16 20:54:01,253 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-16 20:54:01,408 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04032323569698305, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051452526799265944, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04032323569698305, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17566666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 20:54:01,408 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-16 20:54:01,408 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.1808787225182908, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.04032323569698305, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.051452526799265944, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.04032323569698305, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.17566666666666667, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.24775, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-16 20:54:01,697 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 424 ms to try 5000 different iterations.  Found a solution that moves 2779 regions; Going from a computed imbalance of 0.1808787225182908 to a new imbalance of 0.013049608962390555. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022165471868651813); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03264955230923183, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.30877777777777776, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022165471868651813); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0095); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04691666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 20:54:01,706 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1942636744:78 , srv985353476:78 , srv1191706008:81 , srv1374645635:81 , srv1925418933:81 , srv54630959:81 , srv1036894495:82 , srv1401795885:82 , srv1788957987:82 , srv1898409639:82 , srv402621384:82 , srv458885464:82 , srv47534880:82 , srv1408008551:83 , srv1637792304:83 , srv1958144443:83 , srv2060352623:83 , srv2133405805:83 , srv216972212:83 , srv228251024:83 , srv485241821:83 , srv887595064:83 , srv918323524:83 , srv939795244:83 , srv1512139203:84 , srv1718254444:84 , srv1820366166:84 , srv194765176:84 , srv690378883:84 , srv862307642:84 , srv895627113:84 , srv934226878:84 , srv1040259946:85 , srv1051308801:85 , srv1053668713:85 , srv133913198:85 , srv1403002210:85 , srv1573942930:85 , srv1820334161:85 , srv2023026317:85 , srv244026697:85 , srv325833901:85 , srv535740985:85 , srv535948106:85 , srv54841523:85 , srv640695358:85 , srv819151041:85 , srv88129284:85 , srv1182258107:86 , srv1448282540:86 , srv1629122668:86 , srv1837108774:86 , srv1981199181:86 , srv2107071244:86 , srv297574165:86 , srv306712946:86 , srv441171474:86 , srv486891914:86 , srv570214459:86 , srv765700448:86 , srv838706054:86 , srv1081728014:87 , srv1181798119:87 , srv1500361580:87 , srv1611655391:87 , srv1785869427:87 , srv2079400271:87 , srv282377255:87 , srv427085076:87 , srv529032350:87 , srv602563603:87 , srv658294223:87 , srv72901648:87 , srv743974627:87 , srv775015900:87 , srv779505717:87 , srv1011581941:88 , srv1087817474:88 , srv149646235:88 , srv1831766628:88 , srv1918079185:88 , srv524318482:88 , srv840339351:88 , srv855882867:88 , srv1216253621:89 , srv1318718344:89 , srv134326997:89 , srv1396812949:89 , srv2069467738:89 , srv236415718:89 , srv877316175:89 , srv1260771203:90 , srv572871616:90 , srv1104465367:91 , srv613601579:91 , srv733271346:91 , srv894044332:91 , srv1483579262:92 , srv178437805:92 , srv1096490251:523 }
2023-03-16 20:54:01,710 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-16 20:54:01,711 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv458885464=63, srv2079400271=49, srv1820334161=35, srv524318482=67, srv1981199181=45, srv306712946=58, srv149646235=24, srv1820366166=36, srv216972212=52, srv877316175=91, srv178437805=32, srv918323524=96, srv894044332=94, srv1396812949=18, srv1260771203=13, srv1408008551=21, srv690378883=79, srv535948106=70, srv895627113=95, srv1011581941=0, srv855882867=89, srv733271346=81, srv939795244=98, srv1096490251=7, srv819151041=86, srv2133405805=51, srv1483579262=23, srv613601579=76, srv1081728014=5, srv570214459=73, srv1500361580=25, srv1182258107=10, srv1087817474=6, srv297574165=57, srv775015900=84, srv236415718=54, srv54841523=72, srv1925418933=41, srv244026697=55, srv1191706008=11, srv1958144443=44, srv840339351=88, srv486891914=66, srv54630959=71, srv1051308801=3, srv1837108774=38, srv402621384=60, srv1403002210=20, srv1053668713=4, srv985353476=99, srv765700448=83, srv88129284=92, srv1512139203=26, srv427085076=61, srv282377255=56, srv1785869427=33, srv1629122668=29, srv1898409639=39, srv2069467738=48, srv134326997=16, srv1573942930=27, srv194765176=43, srv1374645635=17, srv1181798119=9, srv640695358=77, srv2107071244=50, srv934226878=97, srv1216253621=12, srv441171474=62, srv838706054=87, srv1040259946=2, srv2023026317=46, srv325833901=59, srv72901648=80, srv887595064=93, srv1611655391=28, srv602563603=75, srv1918079185=40, srv228251024=53, srv1448282540=22, srv47534880=64, srv2060352623=47, srv485241821=65, srv1788957987=34, srv535740985=69, srv1637792304=30, srv779505717=85, srv1831766628=37, srv862307642=90, srv1318718344=14, srv1104465367=8, srv1036894495=1, srv1401795885=19, srv572871616=74, srv133913198=15, srv529032350=68, srv1942636744=42, srv743974627=82, srv1718254444=31, srv658294223=78} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-16 20:54:01,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-16 20:54:01,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-16 20:54:01,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-16 20:54:01,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-16 20:54:01,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-16 20:54:01,732 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-16 20:54:01,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-16 20:54:01,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-16 20:54:01,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-16 20:54:01,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-16 20:54:01,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-16 20:54:01,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-16 20:54:01,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-16 20:54:01,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-16 20:54:01,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-16 20:54:01,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-16 20:54:01,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-16 20:54:01,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-16 20:54:01,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-16 20:54:01,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-16 20:54:01,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-16 20:54:01,733 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-16 20:54:01,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-16 20:54:01,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-16 20:54:01,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-16 20:54:01,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-16 20:54:01,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-16 20:54:01,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-16 20:54:01,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-16 20:54:01,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-16 20:54:01,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-16 20:54:01,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-16 20:54:01,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-16 20:54:01,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-16 20:54:01,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-16 20:54:01,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-16 20:54:01,734 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-16 20:54:01,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-16 20:54:01,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-16 20:54:01,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-16 20:54:01,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-16 20:54:01,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-16 20:54:01,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-16 20:54:01,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-16 20:54:01,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-16 20:54:01,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-16 20:54:01,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-16 20:54:01,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-16 20:54:01,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-16 20:54:01,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-16 20:54:01,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-16 20:54:01,735 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-16 20:54:01,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-16 20:54:01,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-16 20:54:01,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-16 20:54:01,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-16 20:54:01,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-16 20:54:01,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-16 20:54:01,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-16 20:54:01,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-16 20:54:01,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-16 20:54:01,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-16 20:54:01,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-16 20:54:01,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-16 20:54:01,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-16 20:54:01,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-16 20:54:01,736 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-16 20:54:01,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-16 20:54:01,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-16 20:54:01,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-16 20:54:01,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-16 20:54:01,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-16 20:54:01,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-16 20:54:01,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-16 20:54:01,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-16 20:54:01,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-16 20:54:01,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-16 20:54:01,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-16 20:54:01,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-16 20:54:01,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-16 20:54:01,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-16 20:54:01,737 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-16 20:54:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-16 20:54:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-16 20:54:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-16 20:54:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-16 20:54:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-16 20:54:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-16 20:54:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-16 20:54:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-16 20:54:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-16 20:54:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-16 20:54:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-16 20:54:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-16 20:54:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-16 20:54:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-16 20:54:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-16 20:54:01,738 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-16 20:54:01,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-16 20:54:01,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-16 20:54:01,739 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-16 20:54:01,874 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022165471868651813); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03264955230923183, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022165471868651813); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0095); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04691666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 20:54:01,874 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-16 20:54:01,874 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.013030149988485566, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.022165471868651813); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.03264955230923183, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.022165471868651813); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0095); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.04691666666666667, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-16 20:54:02,110 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 355 ms to try 5000 different iterations.  Found a solution that moves 997 regions; Going from a computed imbalance of 0.013030149988485566 to a new imbalance of 7.428249761312065E-4. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006898331749076312); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007756102553718168); MoveCostFunction : (multiplier=7.0, imbalance=0.11077777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006898331749076312); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 20:54:02,113 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv2069467738:80 , srv895627113:81 , srv1260771203:82 , srv1396812949:82 , srv1448282540:83 , srv1500361580:83 , srv458885464:83 , srv939795244:83 , srv1216253621:84 , srv134326997:84 , srv1401795885:84 , srv1512139203:84 , srv178437805:84 , srv1788957987:84 , srv2023026317:84 , srv2079400271:84 , srv2107071244:84 , srv2133405805:84 , srv216972212:84 , srv244026697:84 , srv54841523:84 , srv572871616:84 , srv613601579:84 , srv640695358:84 , srv690378883:84 , srv838706054:84 , srv918323524:84 , srv1040259946:85 , srv1081728014:85 , srv1191706008:85 , srv133913198:85 , srv1483579262:85 , srv1637792304:85 , srv1898409639:85 , srv1925418933:85 , srv2060352623:85 , srv47534880:85 , srv485241821:85 , srv486891914:85 , srv535740985:85 , srv535948106:85 , srv602563603:85 , srv765700448:85 , srv855882867:85 , srv862307642:85 , srv934226878:85 , srv985353476:85 , srv1011581941:86 , srv1051308801:86 , srv1053668713:86 , srv1087817474:86 , srv1104465367:86 , srv1318718344:86 , srv1629122668:86 , srv1785869427:86 , srv1837108774:86 , srv1942636744:86 , srv194765176:86 , srv1981199181:86 , srv228251024:86 , srv236415718:86 , srv282377255:86 , srv402621384:86 , srv427085076:86 , srv441171474:86 , srv524318482:86 , srv529032350:86 , srv658294223:86 , srv72901648:86 , srv733271346:86 , srv743974627:86 , srv779505717:86 , srv819151041:86 , srv840339351:86 , srv88129284:86 , srv894044332:86 , srv1181798119:87 , srv1182258107:87 , srv1374645635:87 , srv1403002210:87 , srv1408008551:87 , srv1611655391:87 , srv1718254444:87 , srv1820334161:87 , srv1820366166:87 , srv1831766628:87 , srv1918079185:87 , srv1958144443:87 , srv297574165:87 , srv306712946:87 , srv325833901:87 , srv54630959:87 , srv570214459:87 , srv775015900:87 , srv877316175:87 , srv1036894495:88 , srv149646235:88 , srv1573942930:88 , srv887595064:88 , srv1096490251:548 }
2023-03-16 20:54:02,116 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-16 20:54:02,117 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv458885464=63, srv2079400271=49, srv1820334161=35, srv524318482=67, srv1981199181=45, srv306712946=58, srv149646235=24, srv1820366166=36, srv216972212=52, srv877316175=91, srv178437805=32, srv918323524=96, srv894044332=94, srv1396812949=18, srv1260771203=13, srv1408008551=21, srv690378883=79, srv535948106=70, srv895627113=95, srv1011581941=0, srv855882867=89, srv733271346=81, srv939795244=98, srv1096490251=7, srv819151041=86, srv2133405805=51, srv1483579262=23, srv613601579=76, srv1081728014=5, srv570214459=73, srv1500361580=25, srv1182258107=10, srv1087817474=6, srv297574165=57, srv775015900=84, srv236415718=54, srv54841523=72, srv1925418933=41, srv244026697=55, srv1191706008=11, srv1958144443=44, srv840339351=88, srv486891914=66, srv54630959=71, srv1051308801=3, srv1837108774=38, srv402621384=60, srv1403002210=20, srv1053668713=4, srv985353476=99, srv765700448=83, srv88129284=92, srv1512139203=26, srv427085076=61, srv282377255=56, srv1785869427=33, srv1629122668=29, srv1898409639=39, srv2069467738=48, srv134326997=16, srv1573942930=27, srv194765176=43, srv1374645635=17, srv1181798119=9, srv640695358=77, srv2107071244=50, srv934226878=97, srv1216253621=12, srv441171474=62, srv838706054=87, srv1040259946=2, srv2023026317=46, srv325833901=59, srv72901648=80, srv887595064=93, srv1611655391=28, srv602563603=75, srv1918079185=40, srv228251024=53, srv1448282540=22, srv47534880=64, srv2060352623=47, srv485241821=65, srv1788957987=34, srv535740985=69, srv1637792304=30, srv779505717=85, srv1831766628=37, srv862307642=90, srv1318718344=14, srv1104465367=8, srv1036894495=1, srv1401795885=19, srv572871616=74, srv133913198=15, srv529032350=68, srv1942636744=42, srv743974627=82, srv1718254444=31, srv658294223=78} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-16 20:54:02,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-16 20:54:02,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-16 20:54:02,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-16 20:54:02,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-16 20:54:02,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-16 20:54:02,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-16 20:54:02,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-16 20:54:02,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-16 20:54:02,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-16 20:54:02,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-16 20:54:02,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-16 20:54:02,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-16 20:54:02,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-16 20:54:02,129 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-16 20:54:02,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-16 20:54:02,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-16 20:54:02,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-16 20:54:02,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-16 20:54:02,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-16 20:54:02,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-16 20:54:02,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-16 20:54:02,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-16 20:54:02,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-16 20:54:02,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-16 20:54:02,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-16 20:54:02,130 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-16 20:54:02,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-16 20:54:02,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-16 20:54:02,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-16 20:54:02,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-16 20:54:02,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-16 20:54:02,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-16 20:54:02,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-16 20:54:02,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-16 20:54:02,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-16 20:54:02,131 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-16 20:54:02,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-16 20:54:02,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-16 20:54:02,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-16 20:54:02,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-16 20:54:02,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-16 20:54:02,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-16 20:54:02,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-16 20:54:02,132 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-16 20:54:02,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-16 20:54:02,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-16 20:54:02,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-16 20:54:02,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-16 20:54:02,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-16 20:54:02,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-16 20:54:02,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-16 20:54:02,133 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-16 20:54:02,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-16 20:54:02,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-16 20:54:02,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-16 20:54:02,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-16 20:54:02,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-16 20:54:02,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-16 20:54:02,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-16 20:54:02,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-16 20:54:02,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-16 20:54:02,134 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-16 20:54:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-16 20:54:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-16 20:54:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-16 20:54:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-16 20:54:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-16 20:54:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-16 20:54:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-16 20:54:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-16 20:54:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-16 20:54:02,135 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-16 20:54:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-16 20:54:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-16 20:54:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-16 20:54:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-16 20:54:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-16 20:54:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-16 20:54:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-16 20:54:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-16 20:54:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-16 20:54:02,136 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-16 20:54:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-16 20:54:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-16 20:54:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-16 20:54:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-16 20:54:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-16 20:54:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-16 20:54:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-16 20:54:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-16 20:54:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-16 20:54:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-16 20:54:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-16 20:54:02,137 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-16 20:54:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-16 20:54:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-16 20:54:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-16 20:54:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-16 20:54:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-16 20:54:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-16 20:54:02,138 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-16 20:54:02,244 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006898331749076312); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007756102553718168); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006898331749076312); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 20:54:02,245 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-16 20:54:02,245 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=7.358438329202407E-4, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.006898331749076312); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.007756102553718168); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.006898331749076312); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=2.5E-4); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.004916666666666666); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-16 20:54:02,566 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 420 ms to try 5000 different iterations.  Found a solution that moves 268 regions; Going from a computed imbalance of 7.358438329202407E-4 to a new imbalance of 4.9228955361808866E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.029777777777777778, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 20:54:02,567 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv297574165:82 , srv877316175:82 , srv1260771203:83 , srv1573942930:83 , srv1718254444:83 , srv570214459:83 , srv887595064:83 , srv1011581941:84 , srv1081728014:84 , srv1104465367:84 , srv1191706008:84 , srv133913198:84 , srv134326997:84 , srv1403002210:84 , srv1483579262:84 , srv149646235:84 , srv1611655391:84 , srv1785869427:84 , srv1820366166:84 , srv1831766628:84 , srv1898409639:84 , srv1918079185:84 , srv1925418933:84 , srv1942636744:84 , srv194765176:84 , srv306712946:84 , srv325833901:84 , srv47534880:84 , srv602563603:84 , srv613601579:84 , srv658294223:84 , srv733271346:84 , srv775015900:84 , srv779505717:84 , srv819151041:84 , srv895627113:84 , srv934226878:84 , srv985353476:84 , srv1036894495:85 , srv1040259946:85 , srv1051308801:85 , srv1053668713:85 , srv1181798119:85 , srv1182258107:85 , srv1216253621:85 , srv1318718344:85 , srv1374645635:85 , srv1401795885:85 , srv1408008551:85 , srv1448282540:85 , srv1500361580:85 , srv1629122668:85 , srv1637792304:85 , srv178437805:85 , srv1788957987:85 , srv1820334161:85 , srv1837108774:85 , srv1958144443:85 , srv1981199181:85 , srv2023026317:85 , srv2060352623:85 , srv2069467738:85 , srv2107071244:85 , srv2133405805:85 , srv236415718:85 , srv244026697:85 , srv282377255:85 , srv402621384:85 , srv427085076:85 , srv441171474:85 , srv458885464:85 , srv485241821:85 , srv486891914:85 , srv524318482:85 , srv529032350:85 , srv535740985:85 , srv535948106:85 , srv54630959:85 , srv54841523:85 , srv572871616:85 , srv640695358:85 , srv690378883:85 , srv72901648:85 , srv743974627:85 , srv765700448:85 , srv838706054:85 , srv855882867:85 , srv862307642:85 , srv894044332:85 , srv939795244:85 , srv1087817474:86 , srv1396812949:86 , srv1512139203:86 , srv2079400271:86 , srv216972212:86 , srv228251024:86 , srv840339351:86 , srv88129284:86 , srv918323524:86 , srv1096490251:623 }
2023-03-16 20:54:02,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-16 20:54:02,571 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv458885464=63, srv2079400271=49, srv1820334161=35, srv524318482=67, srv1981199181=45, srv306712946=58, srv149646235=24, srv1820366166=36, srv216972212=52, srv877316175=91, srv178437805=32, srv918323524=96, srv894044332=94, srv1396812949=18, srv1260771203=13, srv1408008551=21, srv690378883=79, srv535948106=70, srv895627113=95, srv1011581941=0, srv855882867=89, srv733271346=81, srv939795244=98, srv1096490251=7, srv819151041=86, srv2133405805=51, srv1483579262=23, srv613601579=76, srv1081728014=5, srv570214459=73, srv1500361580=25, srv1182258107=10, srv1087817474=6, srv297574165=57, srv775015900=84, srv236415718=54, srv54841523=72, srv1925418933=41, srv244026697=55, srv1191706008=11, srv1958144443=44, srv840339351=88, srv486891914=66, srv54630959=71, srv1051308801=3, srv1837108774=38, srv402621384=60, srv1403002210=20, srv1053668713=4, srv985353476=99, srv765700448=83, srv88129284=92, srv1512139203=26, srv427085076=61, srv282377255=56, srv1785869427=33, srv1629122668=29, srv1898409639=39, srv2069467738=48, srv134326997=16, srv1573942930=27, srv194765176=43, srv1374645635=17, srv1181798119=9, srv640695358=77, srv2107071244=50, srv934226878=97, srv1216253621=12, srv441171474=62, srv838706054=87, srv1040259946=2, srv2023026317=46, srv325833901=59, srv72901648=80, srv887595064=93, srv1611655391=28, srv602563603=75, srv1918079185=40, srv228251024=53, srv1448282540=22, srv47534880=64, srv2060352623=47, srv485241821=65, srv1788957987=34, srv535740985=69, srv1637792304=30, srv779505717=85, srv1831766628=37, srv862307642=90, srv1318718344=14, srv1104465367=8, srv1036894495=1, srv1401795885=19, srv572871616=74, srv133913198=15, srv529032350=68, srv1942636744=42, srv743974627=82, srv1718254444=31, srv658294223=78} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-16 20:54:02,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-16 20:54:02,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-16 20:54:02,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-16 20:54:02,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-16 20:54:02,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-16 20:54:02,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-16 20:54:02,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-16 20:54:02,582 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-16 20:54:02,583 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-16 20:54:02,584 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-16 20:54:02,585 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-16 20:54:02,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-16 20:54:02,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-16 20:54:02,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-16 20:54:02,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-16 20:54:02,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-16 20:54:02,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-16 20:54:02,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-16 20:54:02,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-16 20:54:02,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-16 20:54:02,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-16 20:54:02,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-16 20:54:02,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-16 20:54:02,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-16 20:54:02,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-16 20:54:02,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-16 20:54:02,586 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-16 20:54:02,587 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-16 20:54:02,695 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 20:54:02,695 WARN  [Time-limited test] balancer.StochasticLoadBalancer(497): calculatedMaxSteps:720000000 for loadbalancer's stochastic walk is larger than maxSteps:5000. Hence load balancing may not work well. Setting parameter "hbase.master.balancer.stochastic.runMaxSteps" to true can overcome this issue.(This config change does not require service restart)
2023-03-16 20:54:02,695 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=4.735237925294344E-5, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=4.4668347344853864E-4); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0017082358106351274); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=4.4668347344853864E-4); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=4.166666666666667E-4); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=5000
2023-03-16 20:54:02,894 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 298 ms to try 5000 different iterations.  Found a solution that moves 26 regions; Going from a computed imbalance of 4.735237925294344E-5 to a new imbalance of 6.926124944990534E-6. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0028888888888888888); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 20:54:02,894 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1040259946:83 , srv1081728014:83 , srv1087817474:83 , srv1181798119:83 , srv1374645635:83 , srv2107071244:83 , srv228251024:83 , srv862307642:83 , srv1011581941:84 , srv1036894495:84 , srv1051308801:84 , srv1053668713:84 , srv1104465367:84 , srv1182258107:84 , srv1191706008:84 , srv1216253621:84 , srv1318718344:84 , srv1396812949:84 , srv1401795885:84 , srv1408008551:84 , srv1483579262:84 , srv149646235:84 , srv1500361580:84 , srv1512139203:84 , srv1573942930:84 , srv1611655391:84 , srv1629122668:84 , srv1637792304:84 , srv178437805:84 , srv1785869427:84 , srv1788957987:84 , srv1820334161:84 , srv1820366166:84 , srv1831766628:84 , srv1837108774:84 , srv1898409639:84 , srv1918079185:84 , srv1925418933:84 , srv1942636744:84 , srv194765176:84 , srv1958144443:84 , srv1981199181:84 , srv2023026317:84 , srv2060352623:84 , srv2069467738:84 , srv2079400271:84 , srv2133405805:84 , srv216972212:84 , srv236415718:84 , srv244026697:84 , srv282377255:84 , srv297574165:84 , srv306712946:84 , srv325833901:84 , srv402621384:84 , srv427085076:84 , srv441171474:84 , srv458885464:84 , srv47534880:84 , srv485241821:84 , srv486891914:84 , srv529032350:84 , srv535740985:84 , srv535948106:84 , srv54630959:84 , srv54841523:84 , srv570214459:84 , srv572871616:84 , srv602563603:84 , srv613601579:84 , srv640695358:84 , srv658294223:84 , srv690378883:84 , srv72901648:84 , srv733271346:84 , srv743974627:84 , srv765700448:84 , srv775015900:84 , srv779505717:84 , srv819151041:84 , srv838706054:84 , srv840339351:84 , srv855882867:84 , srv88129284:84 , srv887595064:84 , srv894044332:84 , srv895627113:84 , srv918323524:84 , srv934226878:84 , srv939795244:84 , srv985353476:84 , srv1260771203:85 , srv133913198:85 , srv134326997:85 , srv1403002210:85 , srv1448282540:85 , srv1718254444:85 , srv524318482:85 , srv877316175:85 , srv1096490251:684 }
2023-03-16 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-16 20:54:02,898 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv458885464=63, srv2079400271=49, srv1820334161=35, srv524318482=67, srv1981199181=45, srv306712946=58, srv149646235=24, srv1820366166=36, srv216972212=52, srv877316175=91, srv178437805=32, srv918323524=96, srv894044332=94, srv1396812949=18, srv1260771203=13, srv1408008551=21, srv690378883=79, srv535948106=70, srv895627113=95, srv1011581941=0, srv855882867=89, srv733271346=81, srv939795244=98, srv1096490251=7, srv819151041=86, srv2133405805=51, srv1483579262=23, srv613601579=76, srv1081728014=5, srv570214459=73, srv1500361580=25, srv1182258107=10, srv1087817474=6, srv297574165=57, srv775015900=84, srv236415718=54, srv54841523=72, srv1925418933=41, srv244026697=55, srv1191706008=11, srv1958144443=44, srv840339351=88, srv486891914=66, srv54630959=71, srv1051308801=3, srv1837108774=38, srv402621384=60, srv1403002210=20, srv1053668713=4, srv985353476=99, srv765700448=83, srv88129284=92, srv1512139203=26, srv427085076=61, srv282377255=56, srv1785869427=33, srv1629122668=29, srv1898409639=39, srv2069467738=48, srv134326997=16, srv1573942930=27, srv194765176=43, srv1374645635=17, srv1181798119=9, srv640695358=77, srv2107071244=50, srv934226878=97, srv1216253621=12, srv441171474=62, srv838706054=87, srv1040259946=2, srv2023026317=46, srv325833901=59, srv72901648=80, srv887595064=93, srv1611655391=28, srv602563603=75, srv1918079185=40, srv228251024=53, srv1448282540=22, srv47534880=64, srv2060352623=47, srv485241821=65, srv1788957987=34, srv535740985=69, srv1637792304=30, srv779505717=85, srv1831766628=37, srv862307642=90, srv1318718344=14, srv1104465367=8, srv1036894495=1, srv1401795885=19, srv572871616=74, srv133913198=15, srv529032350=68, srv1942636744=42, srv743974627=82, srv1718254444=31, srv658294223=78} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-16 20:54:02,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-16 20:54:02,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-16 20:54:02,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-16 20:54:02,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-16 20:54:02,913 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-16 20:54:02,914 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 30 is on host 30
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 31 is on host 31
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 32 is on host 32
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 33 is on host 33
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 34 is on host 34
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 35 is on host 35
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 36 is on host 36
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 37 is on host 37
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 38 is on host 38
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 39 is on host 39
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 40 is on host 40
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 41 is on host 41
2023-03-16 20:54:02,915 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 42 is on host 42
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 43 is on host 43
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 44 is on host 44
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 45 is on host 45
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 46 is on host 46
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 47 is on host 47
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 48 is on host 48
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 49 is on host 49
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 50 is on host 50
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 51 is on host 51
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 52 is on host 52
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 53 is on host 53
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 54 is on host 54
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 55 is on host 55
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 56 is on host 56
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 57 is on host 57
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 58 is on host 58
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 59 is on host 59
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 60 is on host 60
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 61 is on host 61
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 62 is on host 62
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 63 is on host 63
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 64 is on host 64
2023-03-16 20:54:02,916 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 65 is on host 65
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 66 is on host 66
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 67 is on host 67
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 68 is on host 68
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 69 is on host 69
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 70 is on host 70
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 71 is on host 71
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 72 is on host 72
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 73 is on host 73
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 74 is on host 74
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 75 is on host 75
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 76 is on host 76
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 77 is on host 77
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 78 is on host 78
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 79 is on host 79
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 80 is on host 80
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 81 is on host 81
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 82 is on host 82
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 83 is on host 83
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 84 is on host 84
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 85 is on host 85
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 86 is on host 86
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 87 is on host 87
2023-03-16 20:54:02,917 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 88 is on host 88
2023-03-16 20:54:02,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 89 is on host 89
2023-03-16 20:54:02,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 90 is on host 90
2023-03-16 20:54:02,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 91 is on host 91
2023-03-16 20:54:02,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 92 is on host 92
2023-03-16 20:54:02,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 93 is on host 93
2023-03-16 20:54:02,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 94 is on host 94
2023-03-16 20:54:02,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 95 is on host 95
2023-03-16 20:54:02,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 96 is on host 96
2023-03-16 20:54:02,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 97 is on host 97
2023-03-16 20:54:02,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 98 is on host 98
2023-03-16 20:54:02,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 99 is on host 99
2023-03-16 20:54:02,918 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=1, number of hosts=100, number of racks=4
2023-03-16 20:54:03,025 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=6.744069053831948E-6 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0014982219165849825); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 20:54:03,025 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1040259946:83 , srv1081728014:83 , srv1087817474:83 , srv1181798119:83 , srv1374645635:83 , srv2107071244:83 , srv228251024:83 , srv862307642:83 , srv1011581941:84 , srv1036894495:84 , srv1051308801:84 , srv1053668713:84 , srv1104465367:84 , srv1182258107:84 , srv1191706008:84 , srv1216253621:84 , srv1318718344:84 , srv1396812949:84 , srv1401795885:84 , srv1408008551:84 , srv1483579262:84 , srv149646235:84 , srv1500361580:84 , srv1512139203:84 , srv1573942930:84 , srv1611655391:84 , srv1629122668:84 , srv1637792304:84 , srv178437805:84 , srv1785869427:84 , srv1788957987:84 , srv1820334161:84 , srv1820366166:84 , srv1831766628:84 , srv1837108774:84 , srv1898409639:84 , srv1918079185:84 , srv1925418933:84 , srv1942636744:84 , srv194765176:84 , srv1958144443:84 , srv1981199181:84 , srv2023026317:84 , srv2060352623:84 , srv2069467738:84 , srv2079400271:84 , srv2133405805:84 , srv216972212:84 , srv236415718:84 , srv244026697:84 , srv282377255:84 , srv297574165:84 , srv306712946:84 , srv325833901:84 , srv402621384:84 , srv427085076:84 , srv441171474:84 , srv458885464:84 , srv47534880:84 , srv485241821:84 , srv486891914:84 , srv529032350:84 , srv535740985:84 , srv535948106:84 , srv54630959:84 , srv54841523:84 , srv570214459:84 , srv572871616:84 , srv602563603:84 , srv613601579:84 , srv640695358:84 , srv658294223:84 , srv690378883:84 , srv72901648:84 , srv733271346:84 , srv743974627:84 , srv765700448:84 , srv775015900:84 , srv779505717:84 , srv819151041:84 , srv838706054:84 , srv840339351:84 , srv855882867:84 , srv88129284:84 , srv887595064:84 , srv894044332:84 , srv895627113:84 , srv918323524:84 , srv934226878:84 , srv939795244:84 , srv985353476:84 , srv1260771203:85 , srv133913198:85 , srv134326997:85 , srv1403002210:85 , srv1448282540:85 , srv1718254444:85 , srv524318482:85 , srv877316175:85 , srv1096490251:684 }
2023-03-16 20:54:03,055 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnLargeClusterWithRacks Thread=10 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=315 (was 315), ProcessCount=169 (was 169), AvailableMemoryMB=5066 (was 5571)
2023-03-16 20:54:03,064 INFO  [Time-limited test] hbase.ResourceChecker(147): before: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=10, OpenFileDescriptor=260, MaxFileDescriptor=60000, SystemLoadAverage=315, ProcessCount=169, AvailableMemoryMB=5065
2023-03-16 20:54:03,065 INFO  [Time-limited test] balancer.BaseLoadBalancer(1082): slop=0.0, systemTablesOnMaster=false
2023-03-16 20:54:03,065 INFO  [Time-limited test] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=5000, runMaxSteps=true, stepsPerRegion=800, maxRunningTime=120000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 111077.0 etc.
2023-03-16 20:54:03,073 INFO  [Time-limited test] balancer.BalancerTestBase(547): Mock Cluster : { srv1001127129:84 , srv1037373476:84 , srv1058257978:84 , srv1080036852:84 , srv1136411641:84 , srv1281769038:84 , srv130136251:84 , srv137485579:84 , srv152029050:84 , srv1523500039:84 , srv1563685026:84 , srv1644133262:84 , srv1696870851:84 , srv1763891803:84 , srv1817318129:84 , srv1832825338:84 , srv1843329522:84 , srv1863649318:84 , srv2000438406:84 , srv2105830621:84 , srv227330846:84 , srv25927443:84 , srv411844123:84 , srv548510349:84 , srv646832528:84 , srv688339881:84 , srv700156780:84 , srv743728127:84 , srv920304759:84 , srv1706081947:264 } [srvr=30 rgns=2700 avg=90.0 max=90 min=90]
2023-03-16 20:54:03,074 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-16 20:54:03,075 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv920304759=29, srv1523500039=9, srv700156780=27, srv1763891803=14, srv137485579=7, srv25927443=22, srv2105830621=20, srv1706081947=13, srv646832528=25, srv130136251=6, srv152029050=8, srv1696870851=12, srv1001127129=0, srv1832825338=16, srv1037373476=1, srv2000438406=19, srv1136411641=4, srv411844123=23, srv1281769038=5, srv1058257978=2, srv688339881=26, srv1817318129=15, srv227330846=21, srv1080036852=3, srv1563685026=10, srv548510349=24, srv1863649318=18, srv743728127=28, srv1644133262=11, srv1843329522=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-16 20:54:03,082 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-16 20:54:03,083 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-16 20:54:03,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-16 20:54:03,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-16 20:54:03,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-16 20:54:03,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-16 20:54:03,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-16 20:54:03,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-16 20:54:03,084 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-16 20:54:03,099 INFO  [Time-limited test] balancer.StochasticLoadBalancer(345): Running balancer because at least one server hosts replicas of the same region. function cost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1331003146785086, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 20:54:03,099 INFO  [Time-limited test] balancer.StochasticLoadBalancer(505): Start StochasticLoadBalancer.balancer, initial weighted average imbalance=0.9912612438009706, functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.06666666666666667, need balance); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=1.1331003146785086, need balance); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=1.0, need balance); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=1.0, need balance); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0);  computedMaxSteps=64800000
2023-03-16 20:56:03,087 INFO  [Time-limited test] balancer.StochasticLoadBalancer(553): Finished computing new moving plan. Computation took 120001 ms to try 6845562 different iterations.  Found a solution that moves 1860 regions; Going from a computed imbalance of 0.9912612438009706 to a new imbalance of 4.341332789166274E-5. funtionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.6888888888888889, need balance); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 20:56:03,093 INFO  [Time-limited test] balancer.BalancerTestBase(563): Mock after balance: { srv1001127129:90 , srv1037373476:90 , srv1058257978:90 , srv1080036852:90 , srv1136411641:90 , srv1281769038:90 , srv130136251:90 , srv137485579:90 , srv152029050:90 , srv1523500039:90 , srv1563685026:90 , srv1644133262:90 , srv1696870851:90 , srv1706081947:90 , srv1763891803:90 , srv1817318129:90 , srv1832825338:90 , srv1843329522:90 , srv1863649318:90 , srv2000438406:90 , srv2105830621:90 , srv227330846:90 , srv25927443:90 , srv411844123:90 , srv548510349:90 , srv646832528:90 , srv688339881:90 , srv700156780:90 , srv743728127:90 , srv920304759:90 }
2023-03-16 20:56:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer(1718): Start Generate Balance plan for cluster.
2023-03-16 20:56:03,094 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(334): Hosts are {srv920304759=29, srv1523500039=9, srv700156780=27, srv1763891803=14, srv137485579=7, srv25927443=22, srv2105830621=20, srv1706081947=13, srv646832528=25, srv130136251=6, srv152029050=8, srv1696870851=12, srv1001127129=0, srv1832825338=16, srv1037373476=1, srv2000438406=19, srv1136411641=4, srv411844123=23, srv1281769038=5, srv1058257978=2, srv688339881=26, srv1817318129=15, srv227330846=21, srv1080036852=3, srv1563685026=10, srv548510349=24, srv1863649318=18, srv743728127=28, srv1644133262=11, srv1843329522=17} racks are {rack_1=1, rack_0=0, rack_3=3, rack_2=2}
2023-03-16 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 0 is on host 0
2023-03-16 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 1 is on host 1
2023-03-16 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 2 is on host 2
2023-03-16 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 3 is on host 3
2023-03-16 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 4 is on host 4
2023-03-16 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 5 is on host 5
2023-03-16 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 6 is on host 6
2023-03-16 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 7 is on host 7
2023-03-16 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 8 is on host 8
2023-03-16 20:56:03,097 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 9 is on host 9
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 10 is on host 10
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 11 is on host 11
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 12 is on host 12
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 13 is on host 13
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 14 is on host 14
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 15 is on host 15
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 16 is on host 16
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 17 is on host 17
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 18 is on host 18
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 19 is on host 19
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 20 is on host 20
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 21 is on host 21
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 22 is on host 22
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 23 is on host 23
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 24 is on host 24
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 25 is on host 25
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 26 is on host 26
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 27 is on host 27
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 28 is on host 28
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(362): server 29 is on host 29
2023-03-16 20:56:03,098 DEBUG [Time-limited test] balancer.BaseLoadBalancer$Cluster(378): Number of tables=10, number of hosts=30, number of racks=4
2023-03-16 20:56:03,110 INFO  [Time-limited test] balancer.StochasticLoadBalancer(381): Cluster wide - skipping load balancing because weighted average imbalance=0.0 <= threshold(0.025). If you want more aggressive balancing, either lower hbase.master.balancer.stochastic.minCostNeedBalance from 0.025 or increase the relative multiplier(s) of the specific cost function(s). functionCost=RegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); PrimaryRegionCountSkewCostFunction : (multiplier=500.0, imbalance=0.0); MoveCostFunction : (multiplier=7.0, imbalance=0.0); RackLocalityCostFunction : (multiplier=15.0, imbalance=0.0); TableSkewCostFunction : (multiplier=35.0, imbalance=0.0); RegionReplicaHostCostFunction : (multiplier=100000.0, imbalance=0.0); RegionReplicaRackCostFunction : (multiplier=10000.0, imbalance=0.0); ReadRequestCostFunction : (multiplier=5.0, imbalance=0.0); WriteRequestCostFunction : (multiplier=5.0, imbalance=0.0); MemStoreSizeCostFunction : (multiplier=5.0, imbalance=0.0); StoreFileCostFunction : (multiplier=5.0, imbalance=0.0); 
2023-03-16 20:56:03,110 INFO  [Time-limited test] balancer.BalancerTestBase(570): Mock Final balance: { srv1001127129:90 , srv1037373476:90 , srv1058257978:90 , srv1080036852:90 , srv1136411641:90 , srv1281769038:90 , srv130136251:90 , srv137485579:90 , srv152029050:90 , srv1523500039:90 , srv1563685026:90 , srv1644133262:90 , srv1696870851:90 , srv1706081947:90 , srv1763891803:90 , srv1817318129:90 , srv1832825338:90 , srv1843329522:90 , srv1863649318:90 , srv2000438406:90 , srv2105830621:90 , srv227330846:90 , srv25927443:90 , srv411844123:90 , srv548510349:90 , srv646832528:90 , srv688339881:90 , srv700156780:90 , srv743728127:90 , srv920304759:90 }
2023-03-16 20:56:03,122 INFO  [Time-limited test] hbase.ResourceChecker(175): after: master.balancer.TestStochasticLoadBalancerRegionReplicaWithRacks#testRegionReplicationOnMidClusterWithRacks Thread=9 (was 10), OpenFileDescriptor=260 (was 260), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=162 (was 315), ProcessCount=169 (was 169), AvailableMemoryMB=4675 (was 5065)