There are books that deal with mail servers, like O'Reilly's SendMail "bat" book, but they are geared towards configuring and using mail, not coding your own.
The definitive information on how SMTP works was Postel's RFC821, superceded by RFC2821, which was superceded by RFC5321, updated by RFC7504.
The Request For Comments (RFC) documents are generally pretty clear, since they are the authoritative definions of the services and protocols that they define. You have to like monospaced type, though.
SMTP is actually pretty simple. You can even operate it from a telnet text session, since like most of the original Internet protocols, it runs off text commands. That's in part because in the early days of the Internet, the different computers in the Internet might be using ASCII or EBCDIC, but text is text and easily translated from one codeset to another, but binary can be a real mess, since
word sizes and byte orders vary all over the place.
So in essence, you log into the SMTP server (usually via TCP port 25) with a text command, issue another command that says you're sending an email, send the email (if there are binary attachements, they have to be MIME-encoded into text), end the email with another text marker, repeat as desired until you're done. SMTP only sends mail - to receive mail, you need a POP or IMAP server, which are different protocols, different RFC's, and often different programs as well.
The original SMTP was all straight text. Later versions have added features, most notably encryption, but you can start with the basics and build your way up from there.