I have views where I select nodes and group all results in 3 groups by field. When I select for example a limit to 30 I get results but with the different count in every group (for example 20/7/3). What I can use to select for example 10 most relevant results in every group. I can do it with query alter or maybe need to create some plugin?
Yes, query alter helps. Views flexible pager might serve as an inspiration for the code changes. Do contribute back if you find a solution, it has been requested before.
This problem is usually "solved" by creating different Views displays, as many as you have Groups, and filter each field value in its own display. Then show all 3 displays one after another and use styling to make them seem like they're from the same Views.