It is a matter of a frictionless environment. If you include the friction and wind resistance however minimal it might be compared to that of the earths gravitational pull, then eventually you will stop oscillating about the center of the earth and will come to rest at the center. Also if we're going to take into account the smaller forces, then we should include that the earth is not a perfect sphere. Therefore, there will be pull from odd angles and not just into and out of the tunnel we create. Eventually, you may crash into the side of the tunnel unless it is a rather large tunnel.