In normal targeting mode, you will need to have the reticule in the vicinity of the enemy and hit left trigger to lock on. Normal mode is the default. However, it does prefer to target enemies better than animals/birds. This is to help in a firefight, they don't want you to lock on to a rabbit next to some dude shooting at you.
In casual targeting mode, just hit the left trigger and it will lock on to the nearest enemy. Similar rules apply as above.