You can apply security directly to a document using its publication settings. After uploading the document, open Security and select Use custom security settings.
Select No users or groups, then use the pencil icon to choose the users or groups who need access. You can assign either Manage or Access permissions to each user, or apply the same permission level to everyone selected.
If needed, untick Inherit permissions from [parent area] to prevent the document from inheriting existing permissions.