You must first create the groups within your course, and assign each student to a group. See the faculty guides>Groups>Set up Groups and Groupings for help with this.
Once the students are assigned to a group or grouping, edit the forum settings. Scroll down to the common module settings section, change the group mode to separate groups. If you have assigned the groups to a grouping, select it from the grouping dropdown and save.