[mega-thread] Automator - A forum bot that responds to domain removal requests (and soon more)

The Gallery
1

I’m calling it automator, it fulfills its destiny by automatically answering all domain removal topics that it receives from start. It will also include keywords that can be searched to find domain-removal topics tagged by automator, it’s almost done :stuck_out_tongue:

The code below is what used to do everything.

const fs = require("fs");
const crypto = require("crypto");
const nodemailer = require("nodemailer");

const transport = nodemailer.createTransport({
	host: "smtp.zoho.eu",
	port: 465,
	secure: true,
	auth: {
		user: "redacted",
		pass: "redacted"
	}
});

const createMessage = from => `Hello ${from}!
Welcome (back?) to Glitch!
For domain removals please send an email to [email protected] so they can start processing your domain removal request.
I totally understand that this might be frustrating, but this is the fastest way to get your custom domain removed.
I hope your problem get solved quickly!

Thanks,
Automator.

> **Note**: This is an automated message.

-- ignore this --
keyword-tag-domain-removal`;

const sendMessage = (user, id, from) => console.log("sending", user, id, from) || transport.sendMail({
	from: "'Spam', [email protected]",
	to: user,
	inReplyTo: id,
	text: createMessage(from)
});

const containsDomainRemoval = t => {
	t = t.trim().toLowerCase();
	return t.includes("domain");
};

exports.register = function ()
{
	this.register_hook("mail", "mail");
	this.register_hook("rcpt", "mail");
	this.register_hook("rcpt_ok", "mail");
	this.register_hook("data", "mail");
	this.register_hook("data_post", "mail");
	this.loginfo("Hello world")
}

exports.mail = function (next, con)
{
	if (con.hook === "mail")
	{
		con.transaction.parse_body = true;
	}
	if (con.hook === "data_post")
	{
		const user = con.transaction.mail_from.user + "@" + con.transaction.mail_from.host;
		console.log(con.transaction.body.children[0].bodytext)
		let msg = con.transaction.body.children[0].bodytext.match(/(.|\n)*(?=---)/g);
		if (!msg) return next(OK);
		msg = msg[0];
		if (!msg) return next(OK);
		msg = msg.trim();
		if (msg.length > 2000) return next(OK);
		let id = con.transaction.header.headers["message-id"];
		let sub = con.transaction.header.headers["subject"];
		if (!id) return next(OK);
		if (!sub) return next(OK);
		if (!id[0]) return next(OK);
		if (!sub[0]) return next(OK);
		id = id[0].substring(1, id[0].length - 2);
		sub = sub[0].trim();
		let inReplyTo = con.transaction.header.headers["in-reply-to"];
		inReplyTo = inReplyTo[0].substring(1, inReplyTo[0].length - 2);
		let from = con.transaction.header.headers["from"];
		if (!from) return next(OK);
		from = from[0];
		if (from.includes(" via Glitch Support <[email protected]>\n")) from = from.substring(0, from.length - 48);
		console.log(user, inReplyTo, id);
		console.log(from);
		console.log(sub);
		console.log(msg);
		if (containsDomainRemoval(sub) || containsDomainRemoval(msg))
			sendMessage(user, id, from);
	}
	next(OK);
}
6 Likes
2

Btw, if this is not allowd I will take it down again.

1 Like
3

Omega fail on my side:

image
image2494×308 68.2 KB

@EddiesTech

1 Like
4

(ignore this) domain not working test

5

Must have been all the testing.

6

Yeah :joy:

7

Hope we’ll see it in action tomorrow xd

8

This is a really cool - I was gonna do it with the API, but this is a great way to do it without a token! Can’t wait to see it in action. I wonder how long it will take from seeing a topic and then replying

9

This should be in #the-gallery, will Automator do it automatically? :joy:

10

Disclaimer: if you’re using a Gmail account for Nodemailer, you will have to use Gmail oAuth and simply “allowing less secure apps” for your Gmail account will not work. Pro: no need to enable less secure apps, better security, and no need to enter your password in the pass field (the oAuth token is enough). Cons: too complicated.

Instructions here: https://medium.com/@nickroach_50526/sending-emails-with-node-js-using-smtp-gmail-and-oauth2-316fe9c790a1

From a user concerned about other Gmail users, having struggled with Nodemailer in the past.

11

This seems great!!! I :heart: it!!

12

@ihack2712 how does it know if you want a domain removal.

13

actually the code runs on Haraka, which is a node.js mailing server, and a custom sendmail host (Zoho). Google (or gmail) is out of the picture :wink:

It seems Haraka wasn’t the best choice but I am looking into other free options as well.

14

I’ll be doing a remake of this using puppeteer, because I realize now that the Haraka mail server is over 9 years old, and hasn’t a had a commit in their repo in 5 years.

I’ll be posting along when I have other news :slight_smile:

15

Ok! Can’t wait to see it in action when it’s done! :heart:

16

Very confused as to how its gonna get the email of the person who posted for a domain removal request. Or does it email glitch?

17

I have another account setup @automator, which has maling list mode on. So it in theory should receive an email about everything.

Second I went on to my VPS and created a plugin for the mail server known as Haraka (don’t recommend it btw). And that receives emails, sometimes.

For some reason, it seems to be stuck in the past, because it keeps receiving old emails. Idk where they come from.

Screen Shot 2020-07-24 at 18.28.43
Screen Shot 2020-07-24 at 18.28.433446×2438 1.17 MB

Third, I’m using nodemailer and a 3rd party email to send emails back to discourse.

1 Like
18

ahh i got u. btw delete email sonce processed I run into that a lot.if all doesn’t work, just store processed emails in a DB as you get em and cross check u haven’t seen it before

19

That’s not the problem, I have written a short script to test that theory but that doesn’t work, it seems to receive a lot of emails still, old connections are literally opened to send me old emails. I will use a similar method on the new system, but that won’t be using silly emails. xd

20

Status Update 1:

The bot can now log in to the account.

image
image4096×2558 978 KB

1 Like
21

uh why is it loggin in? thought it was email based. if you have it open the glitch site and scrape it there’s a good chance its gonna get banned

1 Like
22

I am making a new bot, because the email one didn’t work very well. And if it gets banned it would be hilarious, it’s not like the bot is gonna spam the entire forum. It has a very specific task to do.

1 Like
23

you can make discourse bots?

24

Well, there is an API for that, but I don’t have any api keys so I can’t do it for Glitch. I’m however using puppeteer to interact with support.glitch.com website.

1 Like
25

Ya um scraping the site every few seconds for new topics is not the best of ideas.

1 Like
26

It’s actually not going to work like that, i’ll post about it later. However the bot is going to use less resources than an average user.

1 Like
27

What do you guys think about this message?

28

If the user can use custom domains, then they probably aren’t new. I’d just make it “Welcome back to Glitch!”

29

edited. Any thoughts?

30

Nope, I think it’s flawless now.

1 Like
31

Hi there @ihack2712!
^^^
That’s what I do, so the Hello {from}! could be Hello @{from}! so it pings them like I do. This makes sure that it sends them an email or whatever and makes sure they get a notification from your post. I think it’s good practise to tag the user
Eddie

32

Indeed! I’ll add that :wink: Ty

1 Like
33

The average person doesn’t visit glitch every minute to check for domain removal requests lol. How does this scraping system use less resources?

34

The bot can now fetch topics with their messages and the OP. It caches which topics has been dealt with, the only thing left is to actually answer the topics.

35

I’m not scraping anything, I’m using the Discourse API. And I send a request to the endpoint every 5 minutes, an average user will send a request to the forum every 5-15 seconds.

36

I think your wrong here. The average user might do it every 5-15 seconds, but they only do it for a few minutes/hours a day, not 24/7.

37

As long as the device is online and the browser is running in the background it actually will send a request every 5-15 seconds. And for an average user, this is true.

38

Ya um I don’t think anybody is on the forum/has a forum tab open for more than a few hours a day. But discourse supports sending msgs via the api w/o an api key?

39

If you don’t understand how this works you can lookup some recent tutorials about service workers, they run as a thread on your device, independent whether or not you have tabs or windows open.

No they do not, I only use Puppeteer for sending messages. Everything else is fetched through the API.

40

Turns out I reached a quota

image
image1290×284 25.5 KB

I’ll be using my account then for a couple of days.

1 Like
41

So does Puppeteer act like a real user? Do you press the login button and the make post button, or do you just POST a URL?

43

Oof. (i do not like to 20 character limit)

44

Spelling error in the message ^^^

45

Oh shoot! Tysm  (Speed typos :stuck_out_tongue: )

2 Likes
46

I think it’s one of those spelling errors where nobody notices unless you read carefully :mag_right:

1 Like
47

how will the bot know if someone needs a domain removal?

@ihack2712

48

It checks the topic title and first post in the topic. If it contains domain it sends the automated message.

1 Like
49

image

1 Like
50

domain test

51

Expect a message to arrive by me in this topic in 5 minutes.

Also, anyone feel free to create a test topic about domain removals just to see if it actually works on other topics as well

52

Does it do posts or just topics, as the majority of the time it’s just us talking about domain removals, not people asking :joy:

53

It only fetches the first post in every topic:

const t = topics[i];
const postID = (await api.getTopic(t.id)).post_stream.stream[0];
const post = await api.getPost(postID);
t.content = post.raw;
t.username = post.username;

Here’s the JSON output of what the final topic object looks like:

	{
		"id": 29098,
		"title": "Test domain removal post",
		"slug": "test-domain-removal-post",
		"created_at": "2020-07-24T15:05:28.137Z",
		"content": "Please ignore this post, this is my final testing post for checking whether or not Automator is going to work, sorry again.",
		"username": "ihack2712"
	},

Just a quick disclaimer!!

I DO NOT STORE ANY INFORMATION EXCEPT FOR A TOPIC ID TO PREVENT THE BOT FROM RE-POSTING OR TO MAKE A REQUEST AGAINST THE SAME TOPIC TWICE.

image
image496×1546 61.2 KB

1 Like
55

It works!!!

I’ve spent literally 1.5 days on this, and it feels amazing to finally have it working!

1 Like
56

Can I test it out? (stupid 20 char limit)

57

Wdym? Like you can probably create a topic or something

58

What next should I automate on the Glitch Support forum? :joy:

59

Takes about a minute to post :slight_smile: Let’s hope it beats any Regulars! :joy: Works great!
Does it look for ‘domain’ or ‘domain removals’? As some posts are asking how to add them. Maybe add some more context to the post, like:

If you want to add a domain, bla, bla, bla.
If you want to remove a domain, bla, bla, bla.
If this automated message does not relate to your topic, please kindly ignore the post

@disc-boop398 I just tested it, no need

60

The reason it takes time is because I only fetch updates from discourse every 5 minutes, I could fetch them per minute as well. That will still not be as bad as a normal discourse client.

61

I just made a topic, too late. Try getting to me before I do it next time -_-

62

:man_facepalming:

63

Domain removal test

64

Not gonna respond to that.

65

@anon70439135 It only replies to topics.

1 Like
66

Okay, I got it.

1 Like
67

I’m gonna have to take off my 2fa for github now, because I’ll be using that account for a day or two on the bot, I need to start the bot browser in headless mode, because it is really annoying that my mac keeps opening the chromium window :joy: It’ll be up in a couple of minutes.

1 Like
68

Do you ever turn you mac off? Might be worth putting it on a server to keep it 24/7 and any other Support Forum Bots you make :joy:

69

I can actually push it to my vps… Yeah i’mma do that, the next time the bot starts it will be from my vps.

1 Like
70

Oops. Maybe you can’t do this @ihack2712 :open_mouth: :frowning:

@jenn as our awesome Glitch Community Engineer, can I request approval of this bot, as this will really help the community save time sending posts about custom domains :slight_smile:

3 Likes
71

EDIT: For context: this is how the automated reply looks like: Refer to the automator wiki topic

Pretty pretty please <3

image

3 Likes
72

image

3 Likes
73

4 Likes
74

Okay let’s stop with giphs, lol

1 Like
75

4 Likes
76

Where are you guys getting all the gifs?

1 Like
77

Also, going to tweak the title just a bit.

78

Okay, lol. I do not like the 20 character limit.

79

Probally giphy.com?

80

81

Lol,
20 chars

82

I use google, just search up (name) gif

83

84

Okay, now it’s time to stop :joy: Let’s just hope @jenn goes “Awww” to the gifs so the bot can be approved :slight_smile: I mean, who can say no to those :joy:

3 Likes
86

Let’s hope :slight_smile: (the quick brown fox jumps over the lazy dog to reach the 20 character limit)

1 Like
87

Throw some random characters in brackets to get around the limit.

ie <ieru2iowesfiefj> but without backticks

88

Ok. (the quick brown fox jumps over the lazy dog to reach the 20 character limit)

2 Likes
89

Is that a joke? :joy:

90

“The quick brown fox jumps over the lazy dog” is used when previewing fonts because it contains all the characters of the english alphabet.

91

Seems like github doesn’t like my linux server :frowning: I’ll be looking further into this.

92

So I have to verify my linux server to github, I am working on writing a program for that.

94

I can’t breathe! I messed up something :joy:

95

The bot is now up and running, I’ll keep it online until I get an explicit message from the moderation or administration to turn it off. (Current version is 216 lines of javascript code :open_mouth: that’s not bad).

The bot is guaranteed to answer domain removal posts within 1-2 minutes.

I’m also going to implement features:

  • deleting an automated message (can be used by administration and/or moderation, and selected moderators).
  • Changing the topic title to [domain-removal] {title} when the topic is confirmed to be related to domain removals.
  • Adding a feedback page to the bot.
  • Adding an API to the bot.

Happy Glitching!

Edit: The bot will operate under my account for 1-2 days before I switch it to @automator.

Edit 2: Everything seems to be stable and working:

image
image754×1098 81 KB

Message to Glitch: If you want me to take this down I will do so ASAP. Otherwise, I’d ask that we can see how this bot goes for some time, and discuss possibly giving me an api key to make the bot a less load for the discourse server.

P.S. I really want to work closely with Glitch regarding this bot to make sure that both Glitch and the community feels that the bot is trusting and welcoming.

2 Likes
96

Is the bot refreshing that image? :thinking:

97

No that was me :joy: Although, that is not a bad idea! Hahah just kidding.

98

Lol :joy: :joy: (the quick brown fox jumps over the lazy dog to reach the 20 character limit)

99

Hi there @ihack2712!
Maybe you should set up Puppeteer to read posts too and more interaction so the bot can get Regular on the forum and can rename, etc. Would love for it to be a moderator so that it can look official/like it’s a bot.

100

That would be very difficult, which is why I’m hoping Glitch will give me an API key with permissions to do some stuff like a regular can, that way it can look more official and more welcoming.

If I get and API key it will be A LOT easier to continue developing the bot, and I will actually take the time to write clean code with pipelines and so on, so everything can be open-source and look beautiful.

1 Like
101

@EddiesTech or @ihack2712 one of you guys need to change the title to Mega thread

102

Just sent an email to @glitch_support so they will hopefully give us permission :slight_smile:
Email body: https://pastebin.com/hSRQ6jDZ
Eddie

1 Like
103

Ouh oh!! What about making the bot automatically adding [mega-thread]

4 Likes
104

Cool! I will send an email as well explaining a couple of things and ask about some sort of collaboration on this.

1 Like
105

Now I’ve also sent an email (or ticket) to Glitch.

image
image1942×1460 384 KB

1 Like
106

Awesome! Hope they get back soon! I think we’ve got a few minutes before they close :joy:

107

How did you get the dark theme on HappyFox? :thinking:

108

Dark Reader extension :wink:

3 Likes
109

Maybe :joy:

But honestly, this is probably something Glitch has to discuss internally whether or not to allow a single forum member to run an unofficial bot and not letting anyone else do the same. I will understand if Glitch denies my request.

2 Likes
110

Makes sense.

111

I’m so enjoying this thread, and that would be really cool feature. More than 100 posts = [megathread] {title}!

4 Likes
112

And there’s so many test threads, might have to clean up a bit.

4 Likes
113

DB Much? Kinda confused why you got a file for every topic,

114

The files doesn’t contain any data, it is empty, the file is simply just there to indicate that the topic has been dealt with, or basically telling the program not to interact with that topic ever again.

Here’s the code that uses these files:

api.fs = {};
if (!Fs.existsSync("/topics_")) Fs.mkdirSync("/topics_", {recursive: true});
api.fs.add = id => Fs.writeFileSync("/topics_/" + id, "");
api.fs.has = id => Fs.existsSync("/topics_/" + id);

	const do_a_loop = async () => {
		const topics = (await api.getLatestTopics()).filter(topic => !api.fs.has(topic.id));
		
		for (let i = 0; i < topics.length; i++)
		{
			const t = topics[i];
			const postID = (await api.getTopic(t.id)).post_stream.stream[0];
			const post = await api.getPost(postID);
			t.content = post.raw;
			t.username = post.username;
			
			if (topicContainsDomain(t) && !api.fs.has(t.id))
			{
				try
				{
					await sendDomainMessage(browser, page, t);
					api.fs.add(t.id);
				} catch (error)
				{
					console.log("Failed to deliver domain removal message to %s on topic %s", t.username, t.title);
					console.log(error);
					console.log();
				}
			} else
			{
				api.fs.add(t.id);
			}
		}
	};
115

@ihack2712 I dont understand why you have 2 threads for @automator

116

This is the old overpopulated thread, I could probably delete it. I made a second thread which will sorta be a Wiki for Automator, better explaining what it does. This thread was rather meant for discussion regarding how things should be done :wink:

2 Likes