I'm not a security expert, but I'd hazard that the main implication is that if your cloud VM was compromised, the attacker would have your one and only GitHub keypair. It might be best to create a second keypair on the cloud VM, and associate that with your GitHub account. If your VM was ever compromised, you would only have to disable one key pair.
Security of public/private keypairs can be tricky. I avoided discussion of this in the book, because
GitHub's help has a lot to offer on the subject.