Simply hover over a collection and click the X in the upper right corner of the collection's bounding box will delete it from the input sets. Alternatively, you can click into a collection, and there will be a button for eliminating the collection in the border of the dialog box.