... how to notify the node author when new comments are posted on their content??
To my knowledge, the current answer to this question is: not possible yet.
More info
Refer to #d8rules status update November 2017 to get a better understanding of the current status of Rules for D8. Here are some (scary) quotes from it:
... Fago can’t dedicate time required to develop the module further. What Rules is really missing at the moment is development capacity to help finish the last milestone.
... Even if we got more funding today, fago the current principal maintainer of the Rules module wouldn’t have enough capacity to do the work himself.
Start here if you wonder how to contribute.
One never realizes what has been done, only what remains to be done ...
Plan B
You might want to look at the Message module (and the related modules that belong to the Message Stack), which has a pretty stable release already, and which you could use to create similar functionality.
