The idea for the ARK Resource Calculator was born in January 2016 from my desires to learn web development and figure out quickly what resources I needed to build my fort in the game, ARK: Survival Evolved. I played this game all the time with my friends, and always wanted an app to help us craft. There were already other applications out there, but they were either difficult to work with (i.e. spreadsheets) or out of date. I figured building my own calculator would be a great way to provide a better tool for the ARK community and also learn the skills I need to kickstart my career in web development.

With everybody’s feedback, I was inspired to work on a new version of the app: faster and with more features. It was also a great way to apply what I’ve learned in the recent years and solidify my understanding of them. I've been working on this on and off since the end of May 2018. This time, I'm going the patient route.

The new version of the app is made in React. And instead of hosting on Heroku, it’s hosted on Netlify, a wonderful inexpensive all-in-one platform for web projects. It’s easy to set up and manage. I love it, thanks Netlify! (Not sponsored)

Notable Changes

• Items aren’t retrieved from a Mongo database anymore, instead it’s from a JSON file that I cleaned up and compressed to a file about the size of 250kb. A database just seems overkill for the amount of items ARK has and what I think they will be having. Having a small file to download all the items is much faster than a database request(s) for all items.
• Search is much faster now that on load, it calculate and cache the search results on load for up to 2 characters, then caching any new results after that.
• When selecting an item, it’ll add the items by the number of yields. This is more realistic than just 1 because players craft by yields at a time.
• Weight is shown for both items and resources.
• Users can toggle between displaying regular resources and base resources.
• And the most anticipated feature: users can lock resources to prevent calculating their base resources.

Scraping Upgrade

Items weren’t perfect, and there were some cases I didn’t handle such as circular resources. For items that had multiple recipes, it would combine those ingredients giving an inaccurate result. These have now been handled, and by doing so, recipes for items made in the Chemistry Bench are captured.

Aside from that, I’m working with JavaScript promises, which alleviates most of the pain I had from the callback hell. I also got most features for the scraper done within a day. A huge difference from the weeks it took me previously to finish the web scraper.

There are plenty more I’d like to do for the app in terms of optimization, code quality, maintainability, and a progressive web app implementation. Let me know if you have any suggestions.

I built the first version using the MEAN stack (Mongo, Express, Angular, Node). It was a popular stack at the time and I figured it would help me in the future with job opportunities, but it was hell to work with. I didn’t know what I was doing, and I was still in my last year in school, so it took around 3-4 months until I launched it in April 2016. Towards the end of the development, I spent 9 straight hours manually entering data for 400-500 items from the wiki because I was too impatient to write a web scraper and wanted to get my app out quick. I was so excited, I shared it on Reddit! But that was a terrible move because the app was deployed with bugs!

The problem was with how I set up the app via Digital Ocean. I moved to Heroku as it was easier for me at the time to understand deployment. It took about a week to get it set up and stabled. My very first web application was out in the public for people to use. I was proud of myself. Then I started receiving emails…

(and there’s plenty more where those came from)

That day I skipped building a web scraper came back to haunt me. I knew I couldn’t keep adding and updating items forever, so I finally started building my very first web scraper, and it took a few weeks of fine tuning before I got it right. I thought it was going to be straight forward, but there were plenty of problems I faced.

I needed a list of all the items to scrape, so I used the wiki’s categories page to scrape all the item links, but later found out that some of the items had references to resources that I never scraped. In other words, I didn’t scrape all the items I needed so that the items can reference their resources without error. The category page wasn’t enough. So I asked myself, how can I get all the item references that I need? I figured I can just add the resource links to my list of items for each item I’m iterating through if I didn't have them in my list already. And it worked!

Not every item page had the same layout. Some redirected to another page (i.e. Iron Ingot). Some were ambiguous (i.e. Metal Ingot or Scrap Metal Ingot). Some were grouped (i.e. Dye). I figured the redirected links are items with an alternative name. Ambiguous items can be their own items. And I had to scrape all the dyes differently than a normal item page.

After a while, my IP address was blocked from their site because I did a lot of burst scraping, meaning I tried to scrape all the links at once. I emailed their support team and we talked it out and I had to change the way I was scraping. Working with asynchronous Javascript was a pain. This is when I learned about callbacks, and I used that pattern to scrape an item one at a time.

The final process for my web scraper ended up like this: scrape for item links -> iterate through item links and scrape, if there’s a resource not in the list, add it -> check if all items have valid resource references -> update database

I was happy about the results. I finally didn’t have to update every little item, and less users are disappointed with their interaction with my app.

Now (7/28/2019), there are over 3000 unique users visiting this app every month, and it fills me with joy to know there are so many people from all over the world using my app.

And what really motivated me was that people were spending their time to write me feedbacks to improve the application and send me kind words. I don’t want to let them down and I am grateful to receive such responses. This experience is why I love building web applications.

Here are some of the feedbacks I’ve received.

Thank you everybody for your feedback and suggestions. You’ve motivated me to work hard on the next version of the app, and I assure you it’s much better.