This is my final progress update for the Borneo Bulletin discord bot, covering my progress from day 6 to day 10. In these last 5 days of development, I managed to complete all my feature requirements except for the news summarizing.
Here’s what I did:
I realized that all the code for my discord bot was written in one file, which could get messy as I added more code. So, I decided to research how to structure my code and I found some code examples of using “cogs” to manage my discord bot commands. I followed a YouTube tutorial and apply it to my project. Took a bit of time to learn it but it was worth it. It now works perfectly, and it is so much more readable after refactoring my code.
🌎 EASY! Organize Your Messy Code Using COGS! (Ep. 13) - YouTube
In my previous progress update, I also said that my scraper is not scraping the right content. It only scrapes some of the headlines but not all. I review back my code and inspect the website components again then I updated my code to scrape the news articles I wanted. I also scraped the first image from the article as well and use it for the thumbnail when I send the news to my discord text channel.
Only headline highlighted in green is scraped while headline highlighted in red is not.
I then work on sending the news on a text channel every 9 am Brunei time. Luckily, there is already documentation on this and in addition to my code refactoring earlier, I can easily add this feature to my bot. I just need to add the time I want to send the message in the UTC time zone and manually adjust it to 9 am Brunei time. So, I need to schedule the message for 1 am UTC because 1 am UTC is 9 am Brunei time.
🌎 discord.ext.tasks – asyncio.Task helpers (discordpy.readthedocs.io).
UTC is a global time standard that helps computers communicate with each other across different time zones.
At first, I need to hardcode the channel ID of where the bot should post the news on, but I solved this problem on the same day by having the user use the /toggle_scheduled_news command. When the user uses this command, the bot can access the text channel ID on where the user uses that command and store that ID so that the bot knows which text channel it should post the scheduled news on.
Me using /toggle_scheduled_news commands on my text
At the end of the day, I deployed the updated code to my virtual private server, allowing my bot to run 24/7 without the need for my own computer.
It is 9 am and the scheduled news feature works! Didn't do much development today so I just updated the readme section for the Borneo Bulletin Bot GitHub repository with the help of ChatGPT. For the rest of the day, I wrote a blog post documenting my progress from day 1 to day 5.
Borneo Bulletin Bot successfully runs scheduled news at 9 am Brunei time.
At 9 am, the bot sends a bunch of news for the day as expected. But among the 6 headlines, one of them is titled “One-minute garlic pan sauce transforms spicy chicken traybake”. That's when I realized there was another issue with the scraper. Another dive into inspect element and I finally fixed the scraper this time.
The articles highlighted in red was scraped when it is not supposed to. The green and red articles shares the same CSS class name.
I was planning to make my discord bot public on this day so other people can invite the bot to their server. Then I asked myself, what would happen if I tried to fetch articles of the day on one text channel and then ran the same command on another channel while the first command was still running? I tried it and as I expected, I couldn't run the second command while the first one is still running.
I was working on other stuff unrelated to this project in the afternoon and ended my day by enabling the public bot option to my Discord bot. Now other people can invite it to their Discord server.
I worked on the problem the next day. Did some research and asked questions on discord.py discord server. I tried to make the scraper asynchronous by using asyncio, but I found out that the blocking still occurred due to the nature of Selenium's web driver. The functions in the discord bot must run asynchronously so that it can handle multiple requests from multiple users at the same time. In the end, I still couldn't fix this problem and wrote a GitHub issue on it. However, I managed to minimize the impact by preventing the blocking from even happening.
Using the /fetch_article command that I created will run the scraper to fetch the headlines of the day every single time even though the news is only updated once per day on the Borneo Bulletin website. To prevent this, why not fetch the article once per day and store it locally, then use /fetch_article commands to just return the articles that are stored locally? That is exactly what I did. Not only it “almost” fixes the blocking problem, but the bot also responds to the commands immediately as it just needs to return the locally stored articles.
Improve response time by caching headlines of the day. ~2 minutes to instant.
I also work on some quality-of-life improvements like adding “add to server” buttons to the discord bot profile and storing the channel id when the user use /toggle_schedule_news locally so that when in case I want to close my bot (for development purpose), the bot will still remember which channel to send the scheduled news.
Borneo Bulletin Bot profile now has 'Add to Server' button.
Last day of development. I spent the last day adding 1 simple command /help to guide users on how to use the bot. Then I wrote this blog.
Borneo Bulletin Bot /help command response
You may notice that I haven’t done the summarization feature for this bot. The reason is that I assigned this task to a friend, but he is on vacation, so I don’t want to bother him. Why don’t just I make it myself? I want to make this project a collaborative effort as I want to treat this project like a real-world project from a client and in a real-world project, you usually work with another developer to finish a project. I will definitely continue adding that feature in the future, but I will not be updating my progress anymore on my blog because the original plan is to finish this project within 10 working days.
All in all, I feel proud to be able to provide a working product for this project. I learn a lot while doing this and I am excited to work on my next fun project (which I haven’t really decided on).
Happy happy happy
