Walkthrough: How to Encrypt and Decrypt Files with RSA and OpenSSL
A guide for IT certification hands on labs for CompTIA A+, Network+, Security+, Linux+, Cloud+, CySA+, PenTest+, CASP+
The point of this walkthrough is to go step-by-step through the basic process of encrypting and decrypting files using the fundamentals of PKI. This is to help you get a handle on the process and give context to help you get through your certifications. The first part of this guide is for using a single public/private key pair. The second part is for using two sets of corresponding key pairs.
For those of you who want a video walkthrough, take a look just below. Remember that you can contact us to get help with a live instructor or with a partner organization!
Step 0: Assumptions and Prerequisites
You have OpenSSL installed.
MacOS (since about 2000) has OpenSSL installed by default. Some Linux distros have OpenSSL already installed by default and some don’t. Windows does not have OpenSSL installed by default. If you don’t, go to this link to follow the official instructions (https://www.openssl.org/source/) or use Dr. Google to search “How to install OpenSSL on {your OS type/version}”.You know how to open your terminal/command prompt, use the cd command, make directories, view the contents of files (cat, less, more, nano, vim, etc.), and follow instructions.
You can use Windows, but you probably just shouldn’t. Get access to Linux or MacOS. It’s just a way better experience. VirtualBox is free and so is Ubuntu :)
Part 1: Single Public/Private Key Pair
This section is just to get the fundamentals down. We will expand on this in later sections.
Step 1: Make a New Directory and Create Some Plaintext to do the Encryption
The first thing we need to do is to great a blank directory for us to work in to make life easier. You can do this with your File Explorer/Finder or with the command line. It doesn’t really matter what it is called. I’ll call mine “rsa_demo” and use the following commands on Mac/Linux:
# Just running cd on Mac/Linux takes you to the home directory. Skip if you know what you're doing. cd # Get into your Desktop directory cd Desktop # Make a new directory for our project mkdir rsa_demo # Get into the rsa_demo directory cd rsa_demo # Verify the directory is blank. There should be no results here. # Mac/Linux ls # Windows dir ### doing Mac/Linux only from this point onward ### # Create some plaintext for our demo. You can do this the command line way or just open a file with a GUI tool like a normal person. Just get some text in a plaintext.txt file echo "my super secret message" > plaintext.txt # This just takes our text and puts it into a file called plaintext.txt. You can change this to whatever you like. # Verify the file is there ls # see the contents # you can use whatever you'd like to view the files. I'm a Linux nerd and like using less, but you can use cat, vim, nano, or whatever else you want. We're just looking here. less plaintext.txt # If you're unfamiliar with less, you just hit the "q" key to quit out of the screen to return to the terminal
Here is a screenshot of what your terminal should look like if you did it right:
Screenshot of creating plaintext.txt
Screenshot of what running "less plaintext.txt" looks like for the RSA demo
Step 2: Generate the Private Key
Note: Yes, I know there are other security features like password protection and a few other things to make this more proper for production. I really doubt you are going to manually do this in real production systems. We are just focusing on getting the basics down to make it through the exams. Feel free to expand on this once you’ve got it down.
The private key is private for a reason. It is the only real thing that is secure. Do not share real production private keys with anyone. Everything depends on private keys being private. Key management is a different topic, but you’ll get there eventually. There is certainly more than one way to do this, and my way is not the most sophisticated. It is purposely as simple as possible.
If you really want to go down a math rabbit hole, you can learn about modular arithmetic and very large prime numbers. For those of you who care, the fundamental academic work on how this works come from the original Diffie and Hellman (1976) paper here and the original Rivest, Shamir, and Adleman (1978) paper here. For the rest of us who just want to do the button mashing magic, skip over those and follow along with the commands. Hope some of you find that interesting.
Here is the command to generate the private key:
# This is the full command to generate a simple private key openssl genrsa -out private_key.pem 1024 # Verify the file was created ls # See the contents of the file. Again, I like less. You can use whatever text editor/viewer you want. less private_key.pem
Let’s step through each chunk of the private key creation to really understand it.
The “openssl” part is just telling our machine that we want to use the OpenSSL tool. The “genrsa” part means that we want to use OpenSSL’s function to generate an RSA key of some kind. “-out” means that we are telling the command that we want to output something. “private_key.pem” is the name and file extension of the private key we want to make. You can change this to whatever you want, but for sanity I’m leaving it like this. “1024” is the number of bits the private key will be. The more bits you have, the stronger the key will be. The fewer bits you have, the easier to break the key will be. You should never use 1024 bits in production because it’s relatively weak. The only reason I use it here is because pretty much any machine can handle that amount of computation. For production, you should use at lease 3092 or 4096 bits. Remember, we’re keeping things simple here.
Here is a screenshot of what your private key should look like if you did it right (your key will be different than mine):
Screenshot of the private key for the RSA demo
Step 3: Generate the Public Key
We have to generate the public key from the private key. There is a really long winded explanation about the mathematics behind it, so if you want to know make sure you read the papers linked in the previous step. For simplicity, I’m not going to torture you with all that nonsense today.
Here is the command to generate the public key:
# This is the full command to generate the public key openssl rsa -in private_key.pem -pubout > public_key.pem # Verify the file was created ls # See the contents of the file. Again, I like less. You can use whatever text editor/viewer you want. less public_key.pem
Now let’s step through this command to make sure we understand what is happening here.
The “openssl” and “rsa” parts are the same explanation as before. We’re just telling openssl to do something with RSA. The “-in” part tells the our command that we are about to take in something, and we specify that the file we want to take in is the “privatekey.pem” file we created earlier. The “-pubout” part tells the command that we are going to output a public key. Using the “>” character is the Unix (Mac/Linux) way of taking the output of whatever is to the left of the symbol and putting it into a file of our definition, which is the “public_key.pem” file in this case. There are other ways to do this, but this is the simplest way I know of.
Here is a screenshot of what your public key should look like if you did it right (your key will be different than mine):
Screenshot of the public key from the RSA demo
Step 4: Encrypt with the Public Key
Now that we have our plaintext and both keys, we can do the encryption. It is really important to remember that for file encryption, you encrypt with the public key and decrypt with the corresponding private key. That is the opposite of the digital signature process, which we will get to in a follow up article.
The reason we do it this way is because anybody can know your public key. It just isn’t important to keep that secret in any practical sense. When we encrypt a file with a public key, it will be jumbled up nonsense to anyone who obtains it. The only way to go from jumbled up ciphertext is to use the corresponding private key to decrypt the file, which we will do shortly.
Now it is time to actually do the encryption. Here is the command:
# This command generates ciphertext from the plaintext and public key openssl pkeyutl -encrypt -inkey public_key.pem -pubin -in plaintext.txt -out ciphertext.txt # Verify the file was created ls # See the contents of the file. You will probably have to type "y" at the prompt because it is a binary file less ciphertext.txt
Let’s break down the command so we understand all the pieces.
The “openssl” part is again just saying what command we want to use. “pkeyutl” is an updated way for OpenSSL to handle encryption with keys. Note that older documentation will probably use “rsautl” for the utility function, but that is deprecated now. “-encrypt” means that we are telling OpenSSL that we want to encryption something. “-inkey” tells the command that we are taking in a key to be used for the encryption that we defined next as using “public_key.pem”. The “-pubin” part of this command may seem a little odd, but we are just explicitly telling OpenSSL that we are using a public key for the encryption. “-in” tells the function what input we want to take for the encryption, which is our “plaintext.txt” file we created earlier in the walkthrough. We round this out by using “-out” to tell OpenSSL we want to output something which we will call “ciphertext.txt” for our example. You can name it whatever you want - just change it in the command.
Here is a screenshot of what your ciphertext should look like if you did it right (your value will be different than mine):
Screenshot of the ciphertext from the RSA demo
Step 5: Decrypt with the Private Key
Next we need to decrypt our ciphertext with the private key. This principle extends far and wide across computing, so understanding the decryption process is essential.
We are in a place where we had a message we only wanted one person to be able to decrypt and view. So far, we have created the jumbled up nonsense of the cipher text. There are hundreds of ways to send the encrypted files from one place to another, but the end result is decryption with the private key.
Let’s take a look at the command to do the decryption:
# This command decrypts the ciphertext and generates the decrypted file openssl pkeyutl -decrypt -inkey private_key.pem -in ciphertext.txt > decrypted.txt # Verify the file was created ls # See the contents of the file. You will probably have to type "y" at the prompt because it is a binary file less decrypted.txt
Just like with the encryption with the public key, the command here is pretty similar. We still have “openssl” to tell us what command to use and “pkeyutl” for using the encryption and decryption functions. “-decrypt” tells us that we are decrypting something. “-inkey” tells us that the next part of the command defines which key we use, so we take in “private_key.pem” to properly decrypt the file. We then use “-in” to define that the file we want to take in to be decrypted is the “ciphertext.txt” file. We then use a little Unix trick with the “>” to send the results of the left side of the symbol to the “decrypted.txt” file on the right side.
Here is a screenshot of what your terminal should look like if you did it right:
Screenshot of the decrypted text for the RSA demo
When we do a check through our files, we’ll see that the contents of “plaintext.txt” and “decrypted.txt” should be identical. But how can we be sure they are identical?
Step 6: Verify File Integrity with Hashing
Hashing is a simple and effective way to, among other functions, verify the integrity of files. Without getting into the weeds, hashing is a one-way function meaning that a unique output ran through a hashing algorithm should give us a unique string called a “message digest.” The logic here is simple. If the hashes match, the contents of the target data are identical. If the hashes are different in any way, there is something non-identical in the files. In our guide, we will be using this logic in the next step to verify that what we started with (plaintext.txt) is what we ended with (decrypted.txt).
Also note that not all hashing algorithms are created equal and not all of them are reliable. Some algorithms can have issues like collisions where two different inputs results in the same hash output. It’s great to know that for the exams, but we don’t need to worry about those details now.
Let’s run a few commands and compare the outputs:
# Hash our files and compare values. I'll be using md5sum for simplicity, but you can use whatever you want. # Many people also use sha256sum since it is much better overall but produces a longer output md5sum plaintext.txt md5sum ciphertext.txt md5sum decrypted.txt
Here is a screenshot of what it looks like on my end:
Screenshot of the md5 hashes of our files for the RSA demo
What we should notice is that the hashes for “plaintext.txt” and “decrypted.txt” match but “ciphertext.txt” does not. This is because we jumbled up all the text with our public key, so it does result in a completely different file.
Part 2: Using Two Key Pairs
In this part we will take the next step by using two sets of public/private key pairs, which is a much more realistic way of doing things. Many of the descriptions will be abridged since they we will be doing mostly the same process as in Part 1, but new ideas will have the relevant details included.
For our demo, we will be role playing as Person A and Person B. We will create two key pair sets, but we will only actually use Person B’s set. You can do the other step on your own of pretending to be Person B and using the correct keys.
Step 1: Make a New Directory and Create Some Plaintext to do the Encryption
Just like the first time around, we need a place to start. For my example, I am keeping the same directory and original plaintext for simplicity. We are focused on the mechanics of how this works rather than adding complexity for no reason.
Step 2: Generate Both Private Keys
Now that we are using two keys, we are going to generate both private keys in the same directory. In practice, this is usually done with two different people in two different locations with two different machines with an exchange of the corresponding public keys. However, it is a lot simpler to just do everything in one directory for learning. Just realize this process happens extremely fast constantly in real life every second of every day.
Let’s use the same commands as above and change the names of the .pem keys:
# This is the full command to generate a simple private key openssl genrsa -out person_A_private_key.pem 1024 openssl genrsa -out person_B_private_key.pem 1024 # Verify the files were created ls # See the contents of the files. Again, I like less. You can use whatever text editor/viewer you want. less person_A_private_key.pem less person_B_private_key.pem
You can name the private keys whatever you like, but the important part is that they are different and you can distinguish the names.
Step 3: Generate Both Public Keys
This is also basically the same as before. We generate public keys from private keys. Make sure you keep the names straight!
# This is the full command to generate the public key openssl rsa -in person_A_private_key.pem -pubout > person_A_public_key.pem openssl rsa -in person_B_private_key.pem -pubout > person_B_public_key.pem # Verify the files were created ls # See the contents of the file. Again, I like less. You can use whatever text editor/viewer you want. less person_A_public_key.pem less person_B_public_key.pem
Step 4: Encrypt with the Public Key
This is the part where we have to keep from getting mixed up. Directionality matters here. Remember that private keys are never exchanged and always stay private. Public keys can be freely exchanged. In practice, there are too many ways to list to exchange public keys ranging from what we are doing to just emailing them to sophisticated key exchange and escrow systems for enterprises. Simple is good for us here.
The first thing we will do is have Person A send Person B a message encrypted with Person B’s public key. Remember that this is going to jumble up the plaintext into ciphertext that can only be decrypted with the corresponding private key, which is specifically Person B’s private key.
Here’s the procedure:
Person A generates some plaintext
Person A obtains Person B’s public key
Person A uses Person B’s public key to encrypt the plaintext into ciphertext
Person A sends Person B the ciphertext
Person B decrypts the ciphertext with Person B’s private key
Note that we already made some plaintext for the first step and we don’t have to worry about sending keys and files back and forth since we’re just putting everything in the same directory.
Here are the commands:
# This command generates ciphertext from the plaintext and Person B's public key openssl pkeyutl -encrypt -inkey person_B_public_key.pem -pubin -in plaintext.txt -out part2ciphertext.txt # Verify the file was created ls # See the contents of the file. You will probably have to type "y" at the prompt because it is a binary file less part2ciphertext.txt
Step 5: Decrypt with Person B’s Private Key
Let’s pretend that we as Person A sent Person B the part2ciphertext.txt file somehow like an scp copy, ftp transfer, email, etc. We need to switch our perspective to person B now so we can decrypt the message.
The commands are very similar to Part 1:
# This command decrypts the part2ciphertext and generates the part2decrypted file openssl pkeyutl -decrypt -inkey person_B_private_key.pem -in part2ciphertext.txt > part2decrypted.txt # Verify the file was created ls # See the contents of the file. You will probably have to type "y" at the prompt because it is a binary file less part2decrypted.txt
Step 6: Verify File Integrity with Hashing
This part is pretty simple now. We just run our hashing algorithm of choice and make sure the hashes match just like before.
# Hash our files and compare values. I'll be using md5sum for simplicity, but you can use whatever you want. # Many people also use sha256sum since it is much better overall but produces a longer output md5sum part2plaintext.txt md5sum part2ciphertext.txt md5sum part2decrypted.txt
Wrap Up
So, hopefully this gets you going the right direction with basic asymmetric encryption with RSA. By no means is this an exhaustive list of everything that can be done, but truly understanding this process is something I constantly see students struggle with, even at the highest level certifications. It is amazing how much of cybersecurity relies on these basic principles. To a large degree, the only tangible and real technical piece of cybersecurity infrastructure is a handful of private keys for encryption that we all depend on every single day.
For those of you who are looking for additional help or have questions, make sure you contact us to get you going the right direction today!