Ancestry Interview Question

math related problem in hacker rank