$\begingroup$

I need to check this formula I have to compute the distance of a point with latitude $lat$ from the equator:

$$ \mathrm{feet} = \mathrm{lat} * 10000 \times 3280 / 90 $$

Example: A point at $40.480125^\circ N$ has distance

40.480125 * (10000 / 90) * 3280 = 14752756.66666667 ft

from the Equator. My stored value for this is 1417623.

The problem I am having is the numbers are not making sense. I am told I have data that was already converted and is a 7 digit number, but all my calculations end up being an 8 digit number with a decimal.

Why doesn't the result make sense ?

$\endgroup$ 4

1 Answer

$\begingroup$

The formula is taking the circumference of the Earth (assumed to be a perfect sphere) as exactly 40,000 km., which is OK to two significant figures. The arclength along the Earth's surface is then found as

$$ \frac{\text{angle from Equator}}{90º} \ \cdot \ 40,000 \ \text{km.} \ \ , $$

and the result is converted to feet using $ \ \frac{3280 \ \text{ft.}}{1 \ \text{km.}} \ $ , alright to three significant figures. The result is good to two significant figures. Your result for 40.48º North should have eight digits, so someone else slipped up...

$\endgroup$ 7

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy