“Checkout” is a noun. “Check out” is a verb (specifically a “phrasal verb”).
“Setup” is a noun. “Set up” is a verb.
Bad:
“How to setup X”
“First, setup X”
“Checkout the git repo.”
“For more info, checkout the documentation on the wiki.”
Good:
“How to set up X”
“First, set up X”
“Check out the git repo.”
“For more info, check out the documentation on the wiki.”
Bad:
“That’s a great set up.” (You have a great set which is up?)
“Maybe the check out failed.”
Good:
“That’s a great setup.”
“Maybe the checkout failed.”