If a spell is returned to its owner's hand, it's removed from the stack and thus will not resolve. The spell isn't countered; it just no longer exists. This works against a spell that can't be countered.

If a spell with {X} in its mana cost is returned to a player's hand this way, {X} is the value chosen as the spell was cast for the purpose of evaluating the spell's mana value.

If a permanent with {X} in its mana cost is returned to a player's hand this way, {X} is 0 for the purpose of evaluating its mana value.

If you cast a spell "without paying its mana cost," you can't pay any alternative costs. You can, however, pay additional costs, such as kicker costs. If the card has any mandatory additional costs, you must pay those.

If the spell you cast has {X} in its mana cost, you must choose 0 as the value of X.