Hosting my own E-mail server on a NAS/Residential Internet

[Zemekis]

Got a DS420+ NAS about a year ago and I've been taking on projects now & again since then to brush up on tech skills, or just try to do things I think are interesting. I've used email most of my life, but never given too much thought as to how it works until my boss a few years back mentioned that you can host your own e-mail server, though it may be a pain in the ass. I'm fairly privacy-conscious, but more than that, I hate Big Tech. When I heard that Google and the other big guys were openly admitting to training their A.I. using people's inboxes & Google Drive contents, I snapped and started working on my own email hosted on local-hardware I can unplug myself or drop a 10-pound magnet onto.

While Synology NAS boxes running recent DSM operating systems can run an Email Server & Mail Client, I had no idea what I was getting into and was a complete beginner to doing any sort of serious networking beyond some light Port Forwarding here and there years ago. As it turns out, Post 25 (used for SMTP) has been blocked by most residential ISPs since the 90's because it was such a common attack vector. One could easily spam across residential IPs on Port 25 and bombard their open mail port...the issue here is that a block on Port 25 interrupts SMTP email traffic in & out. I can't easily send and receive mail on a residential internet connection. That's where SMTP Outbound Relays and Reflectors come in. They do the same thing, but in different directions...and for some reason have different names (which made this hard to find). You can configure your e-mail server to send mail on a different port than 25, but other servers won't be listening for traffic on whichever random port you decide on...so you use a 'relay'. A service which you configure to listen to the port you choose to send e-mail on, then it passes along your e-mail on Port 25 so that your e-mail becomes deliverable to everyone else. The outbound portion was the easy part...for whatever reason, there are far fewer services willing to act as "reflectors" that will deliver email to my weirdo port selection. In part, this is because on the outbound side of things, only the initial traffic is on a non-standard Port. Once it's shifted over to Port 25, then it's mostly smooth sailing (I'm not even going to get into SPF & SKIM records in this post. There's plenty involved with the initial step, but it's not that bad).

There's no way to let everyone know my non-standard Port so they can send me traffic on that Port. Google isn't going to switch to Port 2525 or whatever just for me. Thus...I paid for a reflector. It captures incoming mail to my domain using my DNS' MX and A Records, then passes them along on Port I chose. It's really just a reverse Outbound Relay, or a glorified forwarding system. What I don't like about this is that it stores & forwards my e-mails. I understand that there isn't really a way around this as it's the industry standard, but I really wish they wouldn't store anything for privacy's sake. I also understand that SMTP e-mail isn't secure and is either stores or transferred in plain-text from what I recall during studying the topic. It's a dumb thing to get privacy-conscious about because it's one of the least privacy-minded methods of communication. 90% of this project was just to see if I could do it, and I'm happy to announce that as of Sunday night, my mail works! Now I need to convince my friends to use it so all my trouble was worth something, haha. 

[Arisien]

Congrats! Do you have a method of automatically updating DNS when your IP changes, or is it static?

 

2 minutes ago, Zemekis said:

I also understand that SMTP e-mail isn't secure and is either stores or transferred in plain-text from what I recall during studying the topic. It's a dumb thing to get privacy-conscious about because it's one of the least privacy-minded methods of communication.

 

SMTP is unencrypted and by default easy prey to man in the middle attacks, but I think the protocol is great in it's simplicity since senders have come up with clever ways to circumvent any lack of features. PGP keys provide you with an anonymous digital signature, allowing you to sign your messages (proving it is you), or encrypt with each others keys so only you can read them. I believe that privacy focused email services like protonmail will just wrap all your messages with a key they automatically generate and then hide the PGP block's prologue and epilogue from the end user. It's also fun to solve other limitations of the protocol, like encoding files in base64 so they can be sent over plaintext.

 

Just now, Zemekis said:

As it turns out, Post 25 (used for SMTP) has been blocked by most residential ISPs since the 90's

 

Yeah, it's unfortunate how many features are blocked by default these days. Have you tried contacting your ISP about it? At least in my experience where I live, the IT guys are pretty nice (far more so than regular customer support). Recently I was helping a friend set up a SSH server on his network and realized he was under a NAT and not directly exposed to the internet. Expecting a hassle and expenses to get him removed from it, I gave his ISP a call and after being transferred over to someone that knows what IPv4 is, I was only one sentence in before he said: "Yeah sure let me just switch you to the other list".

 

22 minutes ago, Zemekis said:

local-hardware I can unplug myself or drop a 10-pound magnet onto.

 

 

[Zemekis]

1 hour ago, Arisien said:

Congrats! Do you have a method of automatically updating DNS when your IP changes, or is it static?

Yeah, it's something called DDNS (Dynamic DNS) that updates if any changes occur. You can set the frequency of the "heartbeat" to limit downtime due to IP changes.

 

1 hour ago, Arisien said:

Yeah, it's unfortunate how many features are blocked by default these days. Have you tried contacting your ISP about it? 

Yeah I did. They open Port 25 for commercial customers (and static IPs) but told me that neither of those are available to a residential customer. It would have saved me a lot of trouble, but I learned a lot finding a workaround!

 

>>>Recently I was helping a friend set up a SSH server on his network and realized he was under a NAT and not directly exposed to the internet.

How did you figure that out?

Thanks for reading my wall of text, I expected crickets in here.

Edited January 17 by Zemekis
Adding a question

[Arisien]

It sucks your ISP won't budge and has you relying on an additional third party to route traffic to you outside your LAN. But it is already in the nature of email to pass through intermediaries for recipients outside of your mail server. Manual encryption is standard for sensitive data.

 

I figured he was under a NAT from the process of elimination. Router was successfully forwarding all LAN traffic to his device, and it was configured on a non standard port so no reason for it to be specifically blocked. ISP-level NATs are standard practice by default nowadays as there is no way something as large as the internet can work with just IPv4, there are simply way too few addresses compared to devices. Long gone are the days of every device having its own IP where people couldn't even imagine the need for more than 256^4 addresses. Since most users don't really care or know if they are under a NAT, it allows the ISP to actually have enough capacity to handle everyone's traffic. Requesting your own public address still seems easy enough, very few people actually need it so at least where I live they just give you it if you ask.

 

Oh, so you do use DDNS. Thought you might have a static IP since you didn't mention it. I myself am freeloading off a friend's subdomain, and since his registrar doesn't offer a DDNS service, wrote a program to periodically curl whatismyip and post an updated entry if there was a change.

[Zemekis]

35 minutes ago, Arisien said:

wrote a program to periodically curl whatismyip and post an updated entry if there was a change.

How/where can I learn to do things like this? There's a lot I want to automate but I don't know where to start.

[Arisien]

Automating web related or just tasks in general? If you have something in mind I can be more specific.

 

In general, it's about getting to know your system. Which OS do you use? It will have a large amount of built in commands you can execute and is packed with utilities that allow you to do a wide range of tasks with the execution of a single command (such as cURL for web client, zlib for de/compression, openssl for cryptography). Getting to know your shell (bash?) will allow you to do more complex operations like chaining them together or piping outputs into each other. You can solve a wide range of tasks with just a short shell script. Then it's about figuring out how to run services on your machine, in my case it's systemd and I can write a simple service file for it which will allow the script to run in the background. Just writing shell scripts will get you pretty far, but at some point it's better to write full programs. Most of these utilities will have a respective C library you can use to integrate into more complex programs you write with the bonus of having easy access to handy features your OS provides.

 

If its web, then I would look into working with HTTP requests. Websites will sometimes provide some level of documented public API for users to be able script with, but you can reverse engineer it yourself using your browsers traffic. Applications like Postman make this task easier since you can record all the requests your browser is making as you use it, replay them programmatically (replicating the behavior of whatever you did manually), and examine specific requests to see their content. This is how people can write programs to automate repetitive simple tasks like updating data on some website, or downloading content hidden from the end user such as videos on sites like twitter. HTTP requests themself are very simple, and really just consist of a type, header and body you fill in and then send with a client.

 

Often times if you need something automated it's very possible someone has already implemented the same thing and you can just use their solution, but there is a charm to solving it yourself

 

 

[Zemekis]

On 1/17/2024 at 8:47 PM, Arisien said:

Automating web related or just tasks in general? If you have something in mind I can be more specific.

 

Which OS do you use?

 

I don't know enough about it in the first place to be specific, my knowledge is kinda spotty. Some IT stuff I know inside and out, then the concepts right next to it I'm entirely ignorant of. I think that comes from being self-taught for the most part & following tutorials/forums/etc. to do complex tasks rather than starting from a really general top-down understanding that something like a Computer Science program would provide. I took a year of Comp Sci in college, did a lot with C++, but they never taught us how to integrate it with anything. We would connect to a UNIX machine and write/compile/run, but they were all math projects, I had no idea how to apply it to anything outside of that box. We never discussed "scripting", scheduled tasks, etc. I learned that was possible just 5 years ago. Where would you go or where did you go to learn about these things? I should go read up instead of wasting your time.

For the sake of example, lets say I want to do what you said: wrote a program to periodically curl whatismyip and post an updated entry if there was a change

 

There is an issue with my DDNS so this would actually be an upgrade. Long story short, I'm doing it through a wizard on my Synology NAS that doesn't account for the registrar I use. Thus, I'm using a solution I found on a FAQ forum held together by hopes & duct tape.

 

In further detail...I can just show you.

Spoiler

The wizard has a dropdown menu of popular domain registrars, but not the one I use (Namecheap). That's not a big issue on its own as they allow me to create a custom provider. However, the password/key field doesn't allow for enough characters. Namecheap's DDNS password is too long. Thus, some guy on the forum provided this as a solution: https://dynamicdns.park-your-domain.com/update?host=__USERNAME__&domain=__HOSTNAME__&password=__PASSWORD__&ip=__MYIP__

I don't know how/why it's working. It's very hard to describe, but the password field does accept all the characters, but there's apparently it won't function if the string is too long. This Query URL allows it to work & it is working.

 

 

I don't really like using someone else's PHP since he could just change it on me as far as I'm aware. Or this park-your-domain.com site could go down and stop hosting the needed code. I'm overall unhappy that I got around my roadblock using a method I don't understand. If I could run something locally to update my DDNS on Namecheap I'd be way more comfortable & I would learn a lot too.

 

I use Windows 10.

Edited Tuesday at 03:52 AM by Zemekis
Image link broke, uploaded a new copy elsewhere

[Arisien]

12 hours ago, Zemekis said:

Thus, some guy on the forum provided this as a solution: https://dynamicdns.park-your-domain.com/update?host=__USERNAME__&domain=__HOSTNAME__&password=__PASSWORD__&ip=__MYIP__

I don't know how/why it's working. It's very hard to describe, but the password field does accept all the characters, but there's apparently it won't function if the string is too long. This Query URL allows it to work & it is working.

 

 

I don't really like using someone else's PHP since he could just change it on me as far as I'm aware. Or this park-your-domain.com site could go down and stop hosting the needed code. I'm overall unhappy that I got around my roadblock using a method I don't understand.

 

That would be your registrar Namecheap's API endpoint for updating your DNS records via HTTP request. This article by them explains it in a bit more detail. It's just a GET request interestingly enough (instead of POST with the data in body), probably to make it easy for non technical people to update it by just opening the URL in their browser instead of using a DDNS updater client. As they say, you can configure a client to use it, but just as an example if you were to automate it with a script (minus a couple lines of error handling for simplicity's sake), this would work:

 

#!/bin/bash

prev_ip="0.0.0.0"

check_ip() {
    current_ip=$(curl -s ifconfig.io)
    if [ "$current_ip" == "$prev_ip" ]; then
        return
    fi
    echo "IP changed to: $current_ip"
    curl -s "https://dynamicdns.park-your-domain.com/update?host=$SRV_HOST&domain=$SRV_DOMAIN&password=$SRV_DDNS_PASS&ip=$current_ip"
    prev_ip="$current_ip"
}

while true; do
    check_ip
    sleep 60
done

 

13 hours ago, Zemekis said:

I use Windows 10.

 

Whoops, probably should have read that before hastily writing above bash xD. But it's simple enough to convert to windows batch or powershell. Not the most common pick for servers, but I imagine it's what your most comfortable with. Funnily enough, Microsoft themselves prefer Linux on their servers. I would look into their documentation for Windows Services to learn how they work and to use them. They should be able to run just about anything you want in the background: an EXE (be it something you installed or your own compiled program) or script in your preferred language. Winget is a new package manager for windows which makes it as easy to install common utilities (like cURL, OpenSSH, git, etc.) with a single command like in Linux, if it's missing a package there are older community driven projects like choco. A lot of these packages could even have a built in service like OpenSSH does.

 

14 hours ago, Zemekis said:

Where would you go or where did you go to learn about these things?

 

I wish I could help more here, but it's such a general topic and I don't really have a great source for everything. There are books available online for learning all sorts of topics from shell scripting, to networking. The internet does have lots of curated content, like this github repo for a list of project-based learning in various programming languages.

 

I guess I just slowly self-taught myself since I was a kid from all over the place. A lot of my technical knowledge originated from modding and messing around with games. I have been studying systems programming and computer architecture at university for some time now.

 

15 hours ago, Zemekis said:

I should go read up instead of wasting your time.

 

I don't mind, it's never a waste for me. If you have any questions I'm happy to help :D.

[Nekone]

I just got to this topic thanks to Koby but I think I got the gist of what was going on. In general I usually discourage operating a mail server from a residential connection because most if not all residential ISPs actually forbid the hosting of any type of server by a residential customer using their network, and this is because depending on the type of server, you'd be not only opening yourself up to attack but also potential abuse - it would not be just you being punished for it, but your ISP would likely face penalties as well ESPECIALLY in the case of SMTP and spam. They thus decided it's never worth the risk, that people can't ever be trusted no matter what they say, and filter the port(s) out at their edge.

 

As for updating the dynamic IP, as you saw most DDNS services have APIs you can call to update it. For example, I utilize Hurricane Electric's DNS services for this - you set up a dynamic hostname there and they have an API URL you can call to update the IP for it. My pfSense+ gateway handles the actual legwork of updating it.

 

There are a multitude of ways to do it - @Arisien's solution would work on most Linux platforms these days (bash = a commonly used shell for Linux systems) for Namecheap's API (though it could easily work for any others that work in a similar manner) and is easy to understand / simple to run. For Windows, as stated yes there are equivalent ways to pull that off but you also have to understand Windows doesn't have most of the tools there to use by default even if you could convert it to Windows batch script or Powershell (though Powershell is designed to be a bit more flexible and might have something that can be useful for that purpose). If it were me, personally - I'd use my IDE (Integrated Development Environment) software to write a small little program to handle a task like that. I could easily design it to run as a Windows Service in the background, even.

 

@Zemekis, since you're using Namecheap's own DDNS API, there's not really much a concern for that to "go anywhere" so I wouldn't worry too much about that, But if you were looking for something a little more robust to work around your NAS' own limitations in that wizard it has, we can explore more alternative solutions for that. You've two people here willing to lend you their expertise, you need only ask.

[Zemekis]

@Nekone @Arisien Thanks fellas, I appreciate the willingness to help/explain. I'm pretty busy at work this week, so this post is more to just let you know I haven't disappeared & I read your posts. What you've both said has given me a fair bit to chew on already, and I have some concepts to read up on before continuing so I can ask intelligent questions. You don't have to restrict answers to Windows, I've used Linux at a beginner level at work & that's what my NAS runs on. I have 2 old laptops that I'm willing to wipe and install Linux on for testing/learning/scheduled processes. I don't want to open up a can of worms here, but what are a few good distros to use as a Novice? My preference would be for one that will set me on the path to using more advanced distros...so if there's one that's beginner friendly but completely unique in its functionality, I wouldn't want that one. I've used Ubuntu for maybe a total of 12 hours...no problems with it but I'm not loyal to it. I hear Mint thrown around pretty often.

 

On 1/18/2024 at 9:41 PM, Nekone said:

In general I usually discourage operating a mail server from a residential connection because most if not all residential ISPs actually forbid the hosting of any type of server by a residential customer using their network, and this is because depending on the type of server, you'd be not only opening yourself up to attack but also potential abuse - it would not be just you being punished for it, but your ISP would likely face penalties as well ESPECIALLY in the case of SMTP and spam.

Completely understandable. Since it's for learning purposes I'm willing to ask for forgiveness instead of permission if it ever comes to that. I get why they block Port 25, if that was wide open then any compromised machine could turn into a 24/7 spam machine. I don't doubt that if someone knew my exact setup and gained access the same could happen to me, but I have a limit of 25 outgoing emails daily anyway. That doesn't discount what you said, but I feel like I'm such a small fry that I'm probably in the clear for small scale academic purposes. Let me think for a while what questions I want to ask and to better define the exact problems I'm trying to solve, because things are working right now.

[Arisien]

3 hours ago, Zemekis said:

I've used Linux at a beginner level at work & that's what my NAS runs on.

 

Oh, I probably should have clarified that was what I was asking. Plenty of people run Windows on their PC's and Linux on their servers.

 

3 hours ago, Zemekis said:

I don't want to open up a can of worms here, but what are a few good distros to use as a Novice? My preference would be for one that will set me on the path to using more advanced distros...so if there's one that's beginner friendly but completely unique in its functionality, I wouldn't want that one. I've used Ubuntu for maybe a total of 12 hours...no problems with it but I'm not loyal to it. I hear Mint thrown around pretty often.

 

Mint is often recommended since it has a very similar experience to Windows. Ubuntu gets used quite commonly on servers and is a nice choice for getting started. If you are looking for a stepping-stone for a more advanced one then maybe Manjaro, a beginner friendly one based on ArchLinux (a minimal distro with a very hands on installation process, which has excellent doc and community). In any case you don't have to worry about locking yourself in with a distro pick. All of these use systemd for init system/service manager, and other than the package manager, pretty much everything else is interchangeable.

 

On 1/19/2024 at 3:41 AM, Nekone said:

In general I usually discourage operating a mail server from a residential connection because most if not all residential ISPs actually forbid the hosting of any type of server by a residential customer using their network, and this is because depending on the type of server, you'd be not only opening yourself up to attack but also potential abuse - it would not be just you being punished for it, but your ISP would likely face penalties as well ESPECIALLY in the case of SMTP and spam. They thus decided it's never worth the risk, that people can't ever be trusted no matter what they say, and filter the port(s) out at their edge.

Wow, didn't know ISP's got THAT strict. Forget running a server at home, here you won't get in trouble for torrenting on a public tracker. Who would penalize them, the IANA?

[Nekone]

On 1/23/2024 at 7:33 PM, Arisien said:

Wow, didn't know ISP's got THAT strict. Forget running a server at home, here you won't get in trouble for torrenting on a public tracker. Who would penalize them, the IANA?

 

We don't NECESSARILY get in trouble for torrenting, unless of course there just so happened to be one of those dumb anti-piracy robots in the swarm that doesn't know what it's doing. Any torrenting I do is rare and usually will be for something like Linux distro images - I like to seed those since I have a really beefy uplink.

 

But I digress - the issue with servers being hosted on residential connections in general is because these can so easily be exploited and abused. SMTP is notorious for this especially, and though the problem was much more rampant in its early days you'd be surprised just how many are still used today for illicit spamming because many inexperienced users will set it up and not secure it properly. In the United States, there's anti-spam legislation in place because of this problem and most reputable residential providers in the US are compelled to adhere to those laws. In addition to that, ISPs just don't want the trouble or the traffic so they automatically take the "all users are dumb idiots" approach and block the default port of 25 at their edge so it will never reach the customer's end of their connection + ignore you when you ask if you can have that opened or state it can't be done.

 

ISPs may also just decide to deny you services if you break their terms by hosting a server on a residential connection - though since money talks, they may just say "if you wanna do this, you must purchase a business package" (which tends to be more expensive).

[Zemekis]

Hello again @Arisien & @Nekone , just checking in to say thanks again for the pointers & that this conversation bore fruit. While the OP was about my email server (still happily chugging along, by the way) I used our following conversation about Linux to finally take the plunge. I'm not an advanced user or even intermediate yet, but I have been using Manjaro for a month with a Plasma desktop as my daily driver at work & loving it. I have a 2nd PC I use at work set up to dualboot Windows 10 (weaning myself off) and Arch. Thankfully my Linux mentor at work is an Arch user and is walking me through it while I use Manjaro solo until I have my Arch experience properly customized. We only get a few hours per week to mess around with my setup, but it's coming along nicely & I have a notebook quickly filling up with notes/cheat-sheets as I learn to use the CLI. The goal is still to get to a point where Arisien's bash script above makes sense & I know how to run/schedule it. I may not be "close" to that yet, but I'm a million miles closer than I was in January. I'll follow up with any interesting milestones or perhaps a question or two. Thanks again.

[Arisien]

12 hours ago, Zemekis said:

Hello again @Arisien & @Nekone , just checking in to say thanks again for the pointers & that this conversation bore fruit. While the OP was about my email server (still happily chugging along, by the way) I used our following conversation about Linux to finally take the plunge. I'm not an advanced user or even intermediate yet, but I have been using Manjaro for a month with a Plasma desktop as my daily driver at work & loving it. I have a 2nd PC I use at work set up to dualboot Windows 10 (weaning myself off) and Arch. Thankfully my Linux mentor at work is an Arch user and is walking me through it while I use Manjaro solo until I have my Arch experience properly customized. We only get a few hours per week to mess around with my setup, but it's coming along nicely & I have a notebook quickly filling up with notes/cheat-sheets as I learn to use the CLI. The goal is still to get to a point where Arisien's bash script above makes sense & I know how to run/schedule it. I may not be "close" to that yet, but I'm a million miles closer than I was in January. I'll follow up with any interesting milestones or perhaps a question or two. Thanks again.

Glad to hear Manjaro/Arch is working out!

 

About running/scheduling, I'll leave an explanation in a spoiler if it's of any use:

Spoiler

To run a shell script you just need to mark the file as executable. On Linux you do this with the chmod command (used for changing file permissions) with the x flag:

 

chmod +x ddns_update.sh

 

Then you can run it the same way you would run any general binary/executable file on the OS

 

./ddns_update.sh

 

Scheduling is done through systemd. The ArchWiki article goes more in depth with configuration options but you could write a ddns_update.service file in the /etc/systemd/system/ folder which looks something like this:

[Unit]
Description=DDNS updater script

[Service]
ExecStart=/home/arisien/bin/ddns_update.sh

[Install]
WantedBy=network.target

 

The location of the script is specified as well as the requirement that it should run after network connection is established.

 

Then you can use systemctl command to manage your services. First reload because the new service file was just added to the folder:

 

sudo systemctl daemon-reload

 

Then specify that the script service should be enabled when the system boots up:

 

sudo systemctl enable ddns_update.service

 

Now the updater script will run automatically.

 

Nice to have you back on the forum \(^ヮ^)/

[Nekone]

On 4/22/2024 at 10:50 PM, Zemekis said:

Hello again @Arisien & @Nekone , just checking in to say thanks again for the pointers & that this conversation bore fruit. While the OP was about my email server (still happily chugging along, by the way) I used our following conversation about Linux to finally take the plunge. I'm not an advanced user or even intermediate yet, but I have been using Manjaro for a month with a Plasma desktop as my daily driver at work & loving it. I have a 2nd PC I use at work set up to dualboot Windows 10 (weaning myself off) and Arch. Thankfully my Linux mentor at work is an Arch user and is walking me through it while I use Manjaro solo until I have my Arch experience properly customized. We only get a few hours per week to mess around with my setup, but it's coming along nicely & I have a notebook quickly filling up with notes/cheat-sheets as I learn to use the CLI. The goal is still to get to a point where Arisien's bash script above makes sense & I know how to run/schedule it. I may not be "close" to that yet, but I'm a million miles closer than I was in January. I'll follow up with any interesting milestones or perhaps a question or two. Thanks again.

Glad to hear it! I'm personally a fan of Debian and Ubuntu and a large number of the servers I operate are of the flavor, but I've drawn some favor towards Arch as well lately. I hope you enjoy it.